class: center, middle, inverse, title-slide # <span style="font-size:48pt;">Logistics & Data Exploration</span> ## 📋 🛠📊 ### Applied Machine Learning in R </br> Pittsburgh Summer Methodology Series ### Lecture 1-B     July 19, 2021 --- class: inverse, center, middle # Overview <style type="text/css"> .remark-code { border: 1px solid grey; } a { background-color: lightblue; } .remark-inline-code { background-color: white; } </style> --- ##Lecture Topics .left-column[ </br> <img src="data:image/png;base64,#agenda.png" width="100%" height="350" /> ] .right-column[ **Brief Refresher on using R** **Packages** - {caret} - primary package for this course - {tidymodels} **Simple Data Split** - Training and testing datasets - Data splitting in {caret} **Exploratory Data Analysis** - Data distributions - Missing data - Feature correlations - Linearity and nonlinearity ] --- class: inverse, center, middle # Brief R Refresher --- ## Software Ecosystem R is like the engine of a car and RStudio is like the cabin - You don't have to use RStudio to use R, but it is more comfortable R packages give you extra features that aren't in R by default - To install a package into your library, use `install.packages("caret")` - To load the {caret} package from your library, use `library("caret")` The {tidyverse} is a collection of popular packages that work together - We will use several functions from {tidyverse} so please make sure to install and load it to follow along - We will try to minimize how much we use it but this may be hard (especially for Jeff) .footnote[ [1] You only need to install a package once per version (e.g., 6.0-88)<br /> [2] You need to load a package once per session (i.e., until closing R) ] --- ## Basic Programming Save objects and give them names using `<-`, then refer to them by name ```r short_pi <- 3.1415 short_pi + 2 #> [1] 5.1415 ``` Numbers can just be typed in R but text data (called **strings**) have to be quoted ```r my_fav_color <- "Cornflower Blue" ``` Use functions to transform inputs into outputs ```r round(2/3) #> [1] 1 ``` Functions often have arguments to let you modify their behavior ```r round(2/3, digits = 3) #> [1] 0.667 ``` --- ## Pipes and Pipelines {tidyverse} gives us the pipe `%>%` which can be read as "AND THEN". It takes an  and passes it to a function as the . ```r library(tidyverse) x <- 4.9823 # Take x and then round it x %>% round() #> [1] 5 ``` We can also use arguments ```r # Take x and then round it to 2 digits, note we don't repeat x inside round() x %>% round(digits = 2) #> [1] 4.98 ``` We can chain these pipes together into a "pipeline" ```r # Take x and then square root it and then round that to 1 digit x %>% sqrt() %>% round(digits = 1) #> [1] 2.2 ``` --- ## Vectors and Subsetting Collect multiple objects of the same type into a single object with `c()` ```r my_numbers <- c(24, 3.5, -10, 1/3) my_numbers #> [1] 24.0000000 3.5000000 -10.0000000 0.3333333 my_friends <- c("Rand", "Perrin", "Matrim", "Egwene") my_friends #> [1] "Rand" "Perrin" "Matrim" "Egwene" ``` Extract parts (i.e., elements) from a vector by positional *index* using `[]` ```r my_numbers[4] #> [1] 0.3333333 my_friends[3] #> [1] "Matrim" ``` Extract multiple elements by giving `[]` a vector ```r my_friends[c(1, 3)] #> [1] "Rand" "Matrim" ``` --- ## Special Data Types If a variable is categorical, we can tell R with `factor()` ```r genre_nums <- c(1, 1, 3, 2, 1, 2, 2, 2, 3) genre_nums #> [1] 1 1 3 2 1 2 2 2 3 genre <- factor(genre_nums, levels = c(1, 2, 3), labels = c("Action", "Comedy", "Drama")) genre #> [1] Action Action Drama Comedy Action Comedy Comedy Comedy Drama #> Levels: Action Comedy Drama ``` If we want to store binary (yes or no, true or false) information, we can use *logicals* ```r my_statements <- c(TRUE, FALSE, FALSE, TRUE) my_statements #> [1] TRUE FALSE FALSE TRUE ``` We can also get logicals by making comparisons ```r my_scores <- c(100, 93, 73, 32, 88, 54) my_scores >= 60 #> [1] TRUE TRUE TRUE FALSE TRUE FALSE ``` --- ## NA for Missing Values We should tell R when a value is missing by setting it to `NA` (not available) ```r bad_data <- c(1.2, 3.2, 2.1, 9999) mean(bad_data) #> [1] 2501.375 good_data <- c(1.2, 3.2, 2.1, NA) mean(good_data) #> [1] NA mean(good_data, na.rm = TRUE) #> [1] 2.166667 ``` We can use {tidyverse} to automatically turn all placeholders into NAs ```r na_if(bad_data, 9999) #> [1] 1.2 3.2 2.1 NA ``` --- ## Data Frames We often collect multiple vectors with the same number of elements into one object This is called a **data frame** where rows are *observations* and columns are *variables* ```r instructors <- data.frame( name1 = c("Jeff", "Shirley", "Aidan"), name2 = c("Girard", "Wang", "Wright"), school = c("KU", "Harvard", "Pitt") ) instructors #> name1 name2 school #> 1 Jeff Girard KU #> 2 Shirley Wang Harvard #> 3 Aidan Wright Pitt ``` We can extract a single variable from a data frame (as a vector) using `$` ```r organizations <- instructors$school organizations #> [1] "KU" "Harvard" "Pitt" ``` --- ## Data Frame Subsetting We can also extract specific rows and columns from a data frame using `[rows, columns]` ```r instructors[2, c(1, 3)] #> name1 school #> 2 Shirley Harvard ``` If we leave rows or columns blank, it will give us all of them ```r instructors[c(1, 2), ] #> name1 name2 school #> 1 Jeff Girard KU #> 2 Shirley Wang Harvard ``` Finally, we can give negative numbers to deselect rather than select ```r instructors[-c(1, 2), ] #> name1 name2 school #> 3 Aidan Wright Pitt ``` --- ## Importing data Often we want to read data into R from external files (e.g., CSV files) In this course, we will be importing data from files stored online (but this also works for local files). We can use the 'read.csv()' function from base R or the `read_csv()` function from {tidyverse} and give it a URL for the file. .scroll-output[ ```r titanic <- read_csv("https://bit.ly/amlr-titanic") titanic #> # A tibble: 963 x 7 #> survived pclass sex age sibsp parch fare #> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> #> 1 yes 1st female 29 0 0 211. #> 2 yes 1st male 0.917 1 2 152. #> 3 no 1st male 30 1 2 152. #> 4 no 1st female 25 1 2 152. #> 5 yes 1st male 48 0 0 26.6 #> 6 yes 1st female 63 1 0 78.0 #> 7 yes 1st female 53 2 0 51.5 #> 8 no 1st male 71 0 0 49.5 #> 9 no 1st male 47 1 0 228. #> 10 yes 1st female 18 1 0 228. #> # … with 953 more rows ``` ] We will also sometimes import data from R packages. In this case, we can use the `data()` function. .scroll-output[ ```r library(psychTools) data(affect) affect #> Study Film ext neur imp soc lie traitanx state1 EA1 TA1 PA1 NA1 EA2 #> 1 maps 3 18.0 9.0 7.0 10.0 3.0 24.0 22.0 24.0 14.0 26.0 2.0 6.0 #> 2 maps 3 16.0 12.0 5.0 8.0 1.0 41.0 40.0 9.0 13.0 10.0 4.0 4.0 #> 3 maps 3 6.0 5.0 3.0 1.0 2.0 37.0 44.0 1.0 14.0 4.0 2.0 2.0 #> 4 maps 3 12.0 15.0 4.0 6.0 3.0 54.0 40.0 5.0 15.0 1.0 0.0 4.0 #> 5 maps 3 14.0 2.0 5.0 6.0 3.0 39.0 67.0 12.0 20.0 7.0 13.0 14.0 #> 6 maps 1 6.0 15.0 2.0 4.0 5.0 51.0 38.0 9.0 14.0 5.0 1.0 7.0 #> 7 maps 1 15.0 12.0 4.0 9.0 3.0 40.0 32.0 1.0 5.0 7.0 0.0 13.0 #> 8 maps 2 18.0 10.0 7.0 9.0 2.0 32.0 41.0 17.0 11.0 10.0 1.0 19.0 #> 9 maps 2 15.0 1.0 3.0 11.0 3.0 22.0 26.0 19.0 5.0 14.0 0.0 19.0 #> 10 maps 2 8.0 10.0 2.0 5.0 2.0 35.0 31.0 15.0 8.0 7.0 0.0 28.0 #> 11 maps 1 13.0 9.0 3.0 9.0 3.0 43.0 39.0 14.0 13.0 10.0 2.0 9.0 #> 12 maps 4 14.0 1.0 3.0 12.0 6.0 33.0 25.0 24.0 15.0 23.0 2.0 27.0 #> 13 maps 4 15.0 2.0 4.0 10.0 5.0 23.0 32.0 7.0 14.0 1.0 0.0 11.0 #> 14 maps 4 19.0 3.0 7.0 11.0 0.0 23.0 23.0 21.0 13.0 20.0 1.0 23.0 #> 15 maps 1 15.0 7.0 4.0 10.0 2.0 27.0 28.0 22.0 15.0 16.0 1.0 16.0 #> 16 maps 1 11.0 13.0 6.0 5.0 7.0 45.0 28.0 2.0 0.0 5.0 0.0 23.0 #> 17 maps 1 16.0 18.0 5.0 10.0 0.0 58.0 56.0 3.0 11.0 3.0 7.0 8.0 #> 18 maps 1 17.0 11.0 6.0 11.0 4.0 39.0 44.0 19.0 18.0 23.0 1.0 21.0 #> 19 maps 1 7.0 10.0 2.0 4.0 1.0 43.0 56.0 14.0 21.0 17.0 8.0 18.0 #> 20 maps 1 13.0 12.0 4.0 8.0 4.0 38.0 35.0 9.0 6.0 1.0 0.0 11.0 #> 21 maps 2 14.0 3.0 5.0 7.0 3.0 35.0 28.0 18.0 8.0 12.0 0.0 16.0 #> 22 maps 2 11.0 10.0 3.0 7.0 1.0 37.0 47.0 11.0 12.0 5.0 1.0 16.0 #> 23 maps 2 20.0 10.0 7.0 10.0 2.0 43.0 43.0 5.0 10.0 2.0 1.0 10.0 #> 24 maps 2 12.0 8.0 1.0 10.0 2.0 36.0 42.0 4.0 15.0 3.0 0.0 11.0 #> 25 maps 2 13.0 9.0 4.0 7.0 1.0 40.0 32.0 14.0 13.0 12.0 1.0 13.0 #> 26 maps 2 10.0 9.0 3.0 7.0 1.0 43.0 38.0 6.0 7.0 4.0 0.0 15.0 #> 27 maps 4 18.0 2.0 6.0 10.0 3.0 30.0 36.0 7.0 10.0 15.0 0.0 10.0 #> 28 maps 4 14.0 12.0 5.0 7.0 4.0 29.0 41.0 12.0 10.0 11.0 1.0 21.0 #> 29 maps 3 13.0 7.0 4.0 7.0 3.0 42.0 42.0 12.0 11.0 7.0 2.0 15.0 #> 30 maps 4 16.0 19.0 6.0 9.0 5.0 35.0 37.0 6.0 13.0 16.0 2.0 19.0 #> 31 maps 4 11.0 14.0 5.0 4.0 2.0 23.0 51.0 13.0 18.0 16.0 10.0 13.0 #> 32 maps 4 8.0 17.0 3.0 5.0 1.0 64.0 61.0 9.0 23.0 9.0 11.0 15.0 #> 33 maps 4 10.0 15.0 5.0 3.0 1.0 40.0 28.0 3.0 12.0 7.0 2.0 20.0 #> 34 maps 3 13.0 14.0 5.0 8.0 2.0 42.0 35.0 24.0 16.0 22.0 4.0 22.0 #> 35 maps 3 11.0 10.0 4.0 5.0 2.0 39.0 32.0 20.0 9.0 12.0 0.0 21.0 #> 36 maps 2 5.0 9.0 1.0 3.0 3.0 46.0 50.0 7.0 12.0 3.0 2.0 19.0 #> 37 maps 3 13.0 10.0 5.0 8.0 2.0 39.0 38.0 0.0 8.0 1.0 3.0 0.0 #> 38 maps 2 7.0 5.0 1.0 3.0 2.0 36.0 34.0 1.0 10.0 0.0 0.0 0.0 #> 39 maps 3 12.0 10.0 4.0 7.0 2.0 38.0 27.0 24.0 18.0 28.0 2.0 18.0 #> 40 maps 2 10.0 13.0 5.0 4.0 0.0 54.0 41.0 6.0 10.0 1.0 6.0 7.0 #> 41 maps 2 14.0 14.0 3.0 10.0 0.0 37.0 45.0 13.0 15.0 17.0 9.0 12.0 #> 42 maps 2 11.0 8.0 3.0 7.0 4.0 29.0 29.0 19.0 9.0 18.0 0.0 16.0 #> 43 maps 1 15.0 2.0 5.0 9.0 5.0 24.0 24.0 23.0 11.0 27.0 2.0 15.0 #> 44 maps 1 14.0 18.0 5.0 7.0 1.0 61.0 28.0 11.0 4.0 10.0 0.0 14.0 #> 45 maps 4 19.0 11.0 7.0 9.0 1.0 34.0 42.0 18.0 19.0 13.0 7.0 23.0 #> 46 maps 1 14.0 13.0 5.0 8.0 3.0 44.0 50.0 5.0 14.0 6.0 6.0 3.0 #> 47 maps 1 11.0 17.0 3.0 8.0 0.0 60.0 62.0 4.0 16.0 6.0 15.0 5.0 #> 48 maps 1 16.0 13.0 5.0 9.0 3.0 45.0 54.0 6.0 12.0 8.0 9.0 5.0 #> 49 maps 1 19.0 13.0 6.0 11.0 2.0 47.0 53.0 6.0 24.0 17.0 11.0 3.0 #> 50 maps 1 20.0 7.0 8.0 10.0 0.0 34.0 31.0 19.0 13.0 15.0 0.0 12.0 #> 51 maps 4 7.0 16.0 0.0 5.0 1.0 46.0 32.0 11.0 12.0 12.0 0.0 21.0 #> 52 maps 1 10.0 18.0 5.0 3.0 2.0 54.0 36.0 15.0 12.0 6.0 1.0 12.0 #> 53 maps 1 9.0 13.0 2.0 6.0 2.0 44.0 33.0 13.0 9.0 15.0 3.0 14.0 #> 54 maps 1 12.0 13.0 4.0 6.0 4.0 48.0 52.0 10.0 16.0 2.0 4.0 9.0 #> 55 maps 3 14.0 12.0 5.0 8.0 1.0 32.0 27.0 19.0 5.0 16.0 0.0 15.0 #> 56 maps 3 20.0 6.0 7.0 12.0 3.0 29.0 25.0 7.0 8.0 8.0 1.0 10.0 #> 57 maps 3 11.0 8.0 3.0 7.0 5.0 38.0 30.0 23.0 17.0 19.0 1.0 18.0 #> 58 maps 4 10.0 10.0 4.0 5.0 2.0 32.0 27.0 2.0 5.0 10.0 0.0 6.0 #> 59 maps 4 14.0 10.0 5.0 8.0 2.0 37.0 29.0 16.0 7.0 14.0 0.0 23.0 #> 60 maps 3 12.0 10.0 4.0 7.0 3.0 36.0 42.0 5.0 12.0 14.0 4.0 4.0 #> 61 maps 3 15.0 3.0 3.0 11.0 2.0 23.0 40.0 8.0 7.0 8.0 0.0 9.0 #> 62 maps 3 12.0 7.0 2.0 9.0 2.0 27.0 24.0 16.0 8.0 21.0 0.0 20.0 #> 63 maps 2 5.0 15.0 3.0 1.0 2.0 57.0 53.0 11.0 11.0 4.0 9.0 12.0 #> 64 maps 4 13.0 6.0 4.0 7.0 3.0 36.0 39.0 21.0 14.0 18.0 2.0 19.0 #> 65 maps 4 10.0 6.0 1.0 9.0 5.0 33.0 31.0 14.0 7.0 9.0 0.0 19.0 #> 66 maps 4 5.0 19.0 3.0 2.0 3.0 51.0 55.0 8.0 16.0 7.0 17.0 16.0 #> 67 maps 4 6.0 14.0 3.0 2.0 4.0 42.0 43.0 7.0 13.0 3.0 1.0 17.0 #> 68 maps 4 18.0 2.0 7.0 9.0 3.0 29.0 32.0 26.0 11.0 27.0 2.0 29.0 #> 69 maps 2 12.0 18.0 4.0 7.0 1.0 50.0 36.0 0.0 7.0 4.0 6.0 10.0 #> 70 maps 2 18.0 13.0 5.0 11.0 3.0 38.0 46.0 12.0 14.0 7.0 5.0 8.0 #> 71 maps 2 17.0 13.0 6.0 10.0 1.0 40.0 51.0 14.0 19.0 20.0 4.0 14.0 #> 72 maps 2 17.0 13.0 6.0 9.0 1.0 51.0 27.0 20.0 13.0 18.0 0.0 24.0 #> 73 maps 2 12.0 5.0 5.0 4.0 0.0 30.0 27.0 23.0 11.0 22.0 0.0 21.0 #> 74 maps 3 6.0 3.0 2.0 2.0 6.0 37.0 27.0 16.0 8.0 10.0 2.0 14.0 #> 75 maps 3 18.0 6.0 7.0 8.0 1.0 35.0 32.0 23.0 13.0 19.0 0.0 23.0 #> 76 maps 3 12.0 16.0 2.0 9.0 0.0 60.0 68.0 8.0 24.0 8.0 23.0 1.0 #> 77 maps 4 17.0 7.0 6.0 10.0 4.0 35.0 61.0 1.0 17.0 0.0 7.0 4.0 #> 78 maps 3 19.0 10.0 8.0 10.0 2.0 38.0 42.0 9.0 12.0 3.0 1.0 6.0 #> 79 maps 4 13.0 18.0 6.0 6.0 2.0 56.0 61.0 16.0 24.0 13.0 8.0 23.0 #> 80 maps 4 13.0 10.0 5.0 8.0 2.0 38.0 40.0 11.0 11.0 9.0 3.0 16.0 #> 81 maps 4 11.0 8.0 3.0 6.0 2.0 42.0 40.0 22.0 16.0 16.0 1.0 21.0 #> 82 maps 4 17.0 13.0 7.0 9.0 0.0 37.0 34.0 20.0 12.0 12.0 3.0 20.0 #> 83 maps 4 19.0 8.0 7.0 9.0 2.0 29.0 26.0 11.0 9.0 12.0 1.0 17.0 #> 84 maps 1 15.0 12.0 6.0 8.0 2.0 30.0 34.0 0.0 10.0 1.0 4.0 14.0 #> 85 maps 3 16.0 0.0 4.0 12.0 5.0 25.0 27.0 14.0 6.0 8.0 0.0 15.0 #> 86 maps 1 13.0 3.0 4.0 8.0 4.0 23.0 35.0 12.0 11.0 5.0 0.0 11.0 #> 87 maps 1 16.0 4.0 6.0 9.0 3.0 44.0 40.0 16.0 16.0 14.0 3.0 12.0 #> 88 maps 2 14.0 10.0 7.0 7.0 2.0 37.0 32.0 10.0 10.0 6.0 1.0 11.0 #> 89 maps 3 13.0 14.0 3.0 8.0 2.0 39.0 63.0 2.0 22.0 8.0 13.0 0.0 #> 90 maps 3 10.0 8.0 2.0 6.0 2.0 39.0 43.0 2.0 10.0 10.0 2.0 1.0 #> 91 maps 3 6.0 5.0 2.0 1.0 1.0 34.0 27.0 1.0 3.0 1.0 0.0 0.0 #> 92 maps 3 2.0 11.0 1.0 1.0 5.0 36.0 36.0 8.0 8.0 9.0 2.0 7.0 #> 93 maps 2 6.0 18.0 1.0 4.0 4.0 71.0 48.0 0.0 10.0 2.0 1.0 23.0 #> 94 maps 2 11.0 15.0 2.0 7.0 1.0 47.0 53.0 0.0 15.0 0.0 3.0 3.0 #> 95 maps 2 19.0 11.0 6.0 11.0 1.0 30.0 40.0 12.0 5.0 13.0 4.0 18.0 #> 96 maps 2 17.0 18.0 8.0 8.0 2.0 71.0 79.0 0.0 25.0 9.0 28.0 12.0 #> 97 maps 4 17.0 12.0 6.0 10.0 1.0 46.0 54.0 5.0 11.0 6.0 6.0 12.0 #> 98 maps 2 10.0 21.0 2.0 8.0 2.0 51.0 70.0 7.0 23.0 5.0 12.0 7.0 #> 99 maps 1 13.0 12.0 4.0 8.0 2.0 43.0 69.0 21.0 26.0 21.0 17.0 10.0 #> 100 maps 1 12.0 7.0 5.0 6.0 5.0 26.0 26.0 16.0 10.0 11.0 1.0 18.0 #> 101 maps 4 14.0 7.0 3.0 9.0 3.0 42.0 37.0 17.0 18.0 20.0 3.0 14.0 #> 102 maps 4 13.0 13.0 5.0 6.0 0.0 39.0 57.0 0.0 19.0 3.0 6.0 0.0 #> 103 maps 1 22.0 10.0 9.0 11.0 0.0 42.0 37.0 0.0 6.0 0.0 0.0 7.0 #> 104 maps 4 8.0 11.0 3.0 4.0 2.0 45.0 56.0 0.0 15.0 0.0 5.0 5.0 #> 105 maps 4 22.0 12.0 8.0 11.0 1.0 44.0 56.0 3.0 19.0 4.0 6.0 19.0 #> 106 maps 4 14.0 9.0 4.0 9.0 5.0 36.0 33.0 7.0 12.0 9.0 2.0 16.0 #> 107 maps 4 9.0 12.0 3.0 6.0 5.0 48.0 46.0 16.0 12.0 13.0 9.0 14.0 #> 108 maps 3 12.0 11.0 5.0 5.0 2.0 37.0 33.0 4.0 7.0 10.0 1.0 3.0 #> 109 maps 3 16.0 3.0 4.0 10.0 2.0 25.0 29.0 2.0 10.0 1.0 0.0 1.0 #> 110 maps 3 13.0 6.0 5.0 7.0 4.0 28.0 34.0 0.0 10.0 1.0 0.0 0.0 #> 111 maps 3 15.0 13.0 3.0 11.0 4.0 38.0 43.0 1.0 14.0 6.0 2.0 1.0 #> 112 maps 3 16.0 10.0 7.0 6.0 1.0 40.0 35.0 12.0 11.0 7.0 0.0 7.0 #> 113 maps 1 14.0 5.0 5.0 8.0 4.0 28.0 26.0 15.0 5.0 19.0 0.0 15.0 #> 114 maps 1 9.0 13.0 2.0 5.0 2.0 40.0 35.0 4.0 14.0 4.0 1.0 9.0 #> 115 maps 2 13.0 10.0 5.0 8.0 2.0 37.0 38.0 26.0 14.0 19.0 3.0 23.0 #> 116 maps 1 21.0 5.0 7.0 12.0 2.0 29.0 39.0 1.0 10.0 4.0 1.0 4.0 #> 117 maps 2 15.0 7.0 4.0 9.0 2.0 35.0 34.0 15.0 13.0 13.0 0.0 15.0 #> 118 maps 2 12.0 2.0 2.0 8.0 6.0 25.0 22.0 17.0 10.0 9.0 0.0 12.0 #> 119 maps 1 17.0 8.0 4.0 11.0 3.0 32.0 25.0 11.0 5.0 17.0 2.0 15.0 #> 120 maps 1 13.0 10.0 5.0 8.0 2.0 53.0 45.0 7.0 7.0 5.0 1.0 14.0 #> 121 maps 1 9.0 5.0 1.0 7.0 2.0 33.0 31.0 7.0 8.0 6.0 1.0 8.0 #> 122 maps 1 19.0 6.0 7.0 9.0 2.0 32.0 55.0 4.0 20.0 2.0 8.0 13.0 #> 123 maps 1 12.0 10.0 3.0 7.0 3.0 39.0 41.0 16.0 12.0 16.0 4.0 12.0 #> 124 maps 1 16.0 7.0 4.0 11.0 4.0 31.0 28.0 18.0 11.0 21.0 1.0 13.0 #> 125 maps 1 10.0 2.0 7.0 2.0 4.0 28.0 23.0 21.0 13.0 28.0 6.0 23.0 #> 126 maps 1 13.0 19.0 5.0 7.0 1.0 54.0 30.0 16.0 10.0 6.0 0.0 16.0 #> 127 maps 1 13.0 11.0 6.0 7.0 2.0 44.0 45.0 4.0 13.0 10.0 3.0 21.0 #> 128 maps 3 13.0 17.0 5.0 8.0 3.0 46.0 54.0 14.0 22.0 10.0 13.0 16.0 #> 129 maps 3 17.0 7.0 6.0 10.0 5.0 32.0 27.0 29.0 17.0 25.0 2.0 10.0 #> 130 maps 4 11.0 20.0 3.0 8.0 3.0 43.0 34.0 13.0 9.0 11.0 1.0 13.0 #> 131 maps 4 14.0 10.0 5.0 8.0 2.0 68.0 60.0 7.0 20.0 2.0 13.0 6.0 #> 132 maps 4 14.0 12.0 5.0 9.0 3.0 39.0 39.0 14.0 19.0 24.0 7.0 11.0 #> 133 maps 3 11.0 10.0 4.0 6.0 3.0 39.0 32.0 9.0 9.0 10.0 0.0 10.0 #> 134 maps 3 10.0 10.0 3.0 6.0 1.0 40.0 32.0 9.0 10.0 10.0 0.0 6.0 #> 135 maps 3 14.0 8.0 5.0 7.0 2.0 31.0 32.0 18.0 15.0 18.0 2.0 14.0 #> 136 maps 3 13.0 11.0 4.0 8.0 2.0 47.0 39.0 11.0 16.0 15.0 10.0 7.0 #> 137 maps 3 15.0 13.0 3.0 11.0 4.0 36.0 48.0 0.0 10.0 7.0 7.0 13.0 #> 138 maps 3 13.0 22.0 5.0 7.0 2.0 52.0 55.0 5.0 16.0 1.0 5.0 7.0 #> 139 maps 1 13.0 2.0 4.0 8.0 2.0 29.0 39.0 8.0 14.0 9.0 3.0 11.0 #> 140 maps 1 20.0 9.0 7.0 11.0 3.0 32.0 36.0 13.0 8.0 9.0 1.0 11.0 #> 141 maps 1 15.0 12.0 5.0 7.0 2.0 34.0 40.0 14.0 14.0 10.0 1.0 16.0 #> 142 maps 1 10.0 18.0 4.0 5.0 4.0 53.0 52.0 2.0 13.0 7.0 2.0 9.0 #> 143 maps 1 22.0 3.0 8.0 12.0 0.0 27.0 38.0 18.0 17.0 18.0 5.0 20.0 #> 144 maps 2 10.0 8.0 2.0 7.0 3.0 34.0 44.0 12.0 11.0 14.0 9.0 17.0 #> 145 maps 2 16.0 13.0 5.0 9.0 2.0 44.0 40.0 17.0 18.0 17.0 1.0 15.0 #> 146 maps 4 16.0 12.0 6.0 7.0 3.0 52.0 48.0 0.0 12.0 0.0 3.0 8.0 #> 147 maps 4 16.0 14.0 5.0 9.0 1.0 38.0 66.0 3.0 24.0 8.0 16.0 8.0 #> 148 maps 4 18.0 12.0 5.0 10.0 3.0 45.0 44.0 3.0 19.0 14.0 4.0 3.0 #> 149 maps 4 18.0 10.0 7.0 10.0 2.0 38.0 39.0 7.0 11.0 5.0 3.0 20.0 #> 150 maps 2 4.0 12.0 0.0 4.0 2.0 53.0 54.0 8.0 15.0 0.0 3.0 8.0 #> 151 maps 2 12.0 2.0 4.0 7.0 5.0 28.0 36.0 6.0 15.0 0.0 0.0 7.0 #> 152 maps 2 21.0 0.0 8.0 12.0 3.0 23.0 23.0 20.0 14.0 21.0 5.0 19.0 #> 153 maps 2 12.0 15.0 4.0 6.0 2.0 45.0 50.0 2.0 16.0 7.0 3.0 17.0 #> 154 maps 2 21.0 9.0 8.0 10.0 2.0 31.0 44.0 6.0 14.0 5.0 0.0 2.0 #> 155 maps 2 11.0 16.0 2.0 8.0 2.0 51.0 47.0 1.0 13.0 7.0 2.0 7.0 #> 156 maps 3 19.0 5.0 5.0 11.0 1.0 33.0 33.0 16.0 17.0 12.0 0.0 12.0 #> 157 maps 3 15.0 13.0 7.0 8.0 4.0 45.0 31.0 18.0 9.0 18.0 4.0 19.0 #> 158 maps 3 1.0 12.0 1.0 0.0 3.0 60.0 55.0 8.0 16.0 1.0 3.0 9.0 #> 159 maps 3 14.0 9.0 6.0 5.0 1.0 41.0 45.0 15.0 13.0 9.0 4.0 27.0 #> 160 maps 3 20.0 16.0 8.0 9.0 0.0 47.0 45.0 2.0 15.0 2.0 6.0 6.0 #> 161 flat 4 7.0 19.0 5.0 2.0 3.0 60.0 45.0 2.0 14.0 2.0 2.0 3.0 #> 162 flat 1 10.0 16.0 3.0 4.0 1.0 52.0 53.0 3.0 18.0 0.0 3.0 7.0 #> 163 flat 1 11.0 7.0 2.0 9.0 3.0 39.0 27.0 21.0 11.0 18.0 1.0 23.0 #> 164 flat 1 10.0 22.0 4.0 4.0 1.0 53.0 68.0 7.0 22.0 6.0 14.0 14.0 #> 165 flat 1 8.0 6.0 1.0 7.0 2.0 38.0 38.0 4.0 8.0 3.0 0.0 2.0 #> 166 flat 4 17.0 14.0 6.0 10.0 1.0 50.0 51.0 1.3 9.0 0.0 9.0 0.0 #> 167 flat 4 17.0 6.0 5.0 11.0 5.0 37.0 36.0 7.0 13.0 7.0 1.0 13.0 #> 168 flat 4 18.0 5.0 5.0 11.0 1.0 30.0 31.0 15.0 13.0 14.0 0.0 20.0 #> 169 flat 4 15.8 6.3 5.0 8.5 1.0 37.0 34.0 0.0 12.0 4.0 0.0 0.0 #> 170 flat 4 7.0 12.0 2.0 4.0 0.0 41.0 45.0 3.0 15.0 4.0 7.0 10.0 #> 171 flat 2 7.0 7.0 2.0 5.0 2.0 35.0 42.0 3.0 14.0 7.0 4.0 2.0 #> 172 flat 3 10.0 11.0 2.0 8.0 3.2 47.0 24.0 20.0 8.0 21.0 0.0 19.0 #> 173 flat 2 12.0 3.0 3.0 7.0 6.0 27.0 25.0 9.0 6.0 7.0 0.0 15.0 #> 174 flat 3 19.0 11.0 7.0 11.0 1.0 41.0 41.0 6.0 13.0 11.0 6.0 14.0 #> 175 flat 1 16.0 7.0 4.0 10.0 2.0 32.0 32.0 0.0 8.0 10.0 0.0 10.0 #> 176 flat 2 5.0 21.0 2.0 2.0 3.0 56.0 51.0 0.0 17.0 1.5 5.0 7.0 #> 177 flat 2 20.0 11.0 7.0 11.0 1.0 45.0 43.0 5.0 13.0 5.0 4.0 11.0 #> 178 flat 2 12.0 14.0 5.0 7.0 1.0 52.0 42.0 8.0 14.0 5.0 5.0 15.0 #> 179 flat 1 7.0 3.0 2.0 3.0 3.0 43.0 56.0 8.0 17.0 4.0 12.0 4.0 #> 180 flat 1 16.0 13.0 6.0 8.0 2.0 40.0 45.0 0.0 14.0 1.0 3.0 6.0 #> 181 flat 1 12.0 12.0 4.0 7.0 3.0 36.0 42.0 13.0 16.0 18.0 8.0 15.0 #> 182 flat 2 12.0 8.0 3.0 8.0 3.0 36.0 34.0 6.0 10.0 9.0 1.0 7.0 #> 183 flat 3 19.0 5.0 7.0 10.0 4.0 28.0 29.0 4.0 3.0 8.0 0.0 3.0 #> 184 flat 3 18.0 7.0 7.0 9.0 1.0 30.0 35.0 16.0 7.0 11.0 2.0 6.0 #> 185 flat 3 17.0 15.0 7.0 8.0 4.0 46.0 41.0 17.0 13.0 20.0 6.0 0.0 #> 186 flat 3 15.0 12.0 4.0 10.0 0.0 37.0 44.0 11.0 11.0 7.0 9.0 7.0 #> 187 flat 3 14.0 12.0 2.0 11.0 0.0 39.0 40.0 1.0 11.0 9.0 9.0 1.0 #> 188 flat 2 14.0 9.0 5.0 8.0 1.0 43.0 40.0 0.0 11.0 0.0 3.0 7.0 #> 189 flat 2 15.0 15.0 6.0 7.0 1.0 38.0 43.0 5.0 14.0 2.0 3.0 5.0 #> 190 flat 2 13.0 18.0 3.0 8.0 0.0 49.0 30.0 9.0 10.0 7.0 0.0 1.0 #> 191 flat 2 14.0 14.0 5.0 8.0 0.0 48.0 50.0 1.0 16.0 0.0 2.0 2.5 #> 192 flat 1 18.0 17.0 8.0 8.0 1.0 44.0 41.0 9.0 15.0 10.0 7.0 9.0 #> 193 flat 4 12.0 10.0 4.0 7.0 2.0 35.0 39.0 11.0 12.0 9.0 0.8 16.0 #> 194 flat 4 21.0 13.0 8.0 12.0 0.0 46.0 50.0 18.0 16.0 13.0 9.0 9.0 #> 195 flat 4 10.0 4.0 4.0 5.0 7.0 35.0 42.0 23.0 14.0 18.0 2.0 17.0 #> 196 flat 4 13.0 2.0 3.0 8.0 0.0 35.0 40.0 9.0 10.0 12.0 5.0 13.5 #> 197 flat 1 19.0 16.0 5.0 13.0 0.0 36.0 34.0 9.0 13.0 8.0 6.0 5.0 #> 198 flat 1 12.0 10.0 4.0 7.0 1.0 34.0 51.0 12.0 18.0 12.0 7.0 9.0 #> 199 flat 1 18.1 9.1 7.1 10.0 0.0 39.0 39.0 8.0 11.0 6.0 1.0 7.0 #> 200 flat 1 7.0 13.0 2.0 3.0 4.0 48.0 62.0 27.0 26.0 18.0 8.0 11.0 #> 201 flat 1 11.0 13.0 3.0 5.0 2.0 39.0 42.0 4.0 10.0 5.0 5.0 0.0 #> 202 flat 3 7.0 20.0 3.0 3.0 2.0 46.0 37.0 8.0 9.0 7.0 1.0 1.0 #> 203 flat 3 9.0 5.0 0.0 9.0 4.0 35.0 30.4 16.0 12.0 5.0 0.0 5.0 #> 204 flat 3 4.0 17.0 1.0 1.0 3.0 53.0 76.0 1.0 24.0 12.0 23.0 17.0 #> 205 flat 3 11.0 4.0 2.0 9.0 3.0 29.0 47.0 9.0 16.0 9.0 5.0 9.0 #> 206 flat 3 11.0 10.0 5.0 6.0 3.0 42.0 46.0 7.1 16.0 10.0 4.0 13.0 #> 207 flat 4 11.0 10.0 2.0 8.0 3.0 38.0 41.0 5.0 15.0 0.0 1.0 13.0 #> 208 flat 4 20.0 1.0 6.0 11.0 2.0 29.0 27.0 22.0 15.0 22.0 1.0 20.0 #> 209 flat 4 12.0 5.0 3.0 9.0 2.2 27.0 36.0 20.0 18.5 16.0 2.0 16.0 #> 210 flat 4 19.0 0.0 7.0 11.0 2.0 28.0 43.0 19.0 15.0 13.0 0.0 14.0 #> 211 flat 1 15.0 4.0 4.0 10.0 3.0 29.0 30.0 11.0 16.0 7.0 0.0 8.0 #> 212 flat 1 19.0 5.0 7.0 10.0 3.0 28.0 30.0 1.0 10.0 6.0 0.0 4.0 #> 213 flat 1 14.0 7.0 5.0 9.0 2.0 30.0 36.0 4.0 10.0 8.0 0.0 2.0 #> 214 flat 1 9.0 11.0 3.0 6.0 3.0 43.0 42.0 4.0 10.0 4.0 0.0 7.0 #> 215 flat 1 19.0 10.0 5.0 12.0 1.0 29.0 32.0 12.0 13.0 22.0 4.0 14.0 #> 216 flat 3 9.0 16.0 3.0 5.0 0.0 56.0 50.0 5.0 10.0 12.0 8.0 17.0 #> 217 flat 2 1.0 15.0 0.0 1.0 3.0 56.0 44.0 6.0 10.0 6.0 2.0 9.0 #> 218 flat 2 10.0 12.0 4.0 5.0 1.0 42.0 56.0 1.0 15.0 8.0 8.0 0.0 #> 219 flat 2 16.0 16.0 6.0 9.0 3.0 43.0 39.0 5.0 11.0 10.0 1.0 4.0 #> 220 flat 4 15.0 17.0 5.0 9.0 2.0 41.0 41.0 2.0 9.0 2.0 3.0 14.0 #> 221 flat 2 19.0 5.0 6.0 11.0 1.0 31.0 48.0 7.0 16.0 8.0 5.0 13.0 #> 222 flat 2 5.0 9.0 0.0 3.0 1.0 34.0 36.0 0.0 8.0 0.0 0.0 0.0 #> 223 flat 4 16.0 12.0 5.0 9.0 2.0 50.0 60.0 7.0 20.0 15.0 18.0 17.0 #> 224 flat 3 4.0 16.0 3.0 1.0 2.0 55.0 59.0 8.0 19.0 0.0 12.0 9.0 #> 225 flat 3 14.0 11.0 5.0 8.0 4.0 34.0 42.0 0.0 18.0 5.0 8.0 11.0 #> 226 flat 3 11.0 6.0 2.0 8.0 3.0 44.0 49.0 5.0 13.0 3.0 3.0 1.0 #> 227 flat 4 13.0 13.0 6.0 6.0 0.0 44.0 53.0 5.0 19.0 1.0 3.1 12.0 #> 228 flat 3 12.0 9.0 3.0 9.0 3.0 43.0 40.0 7.0 13.0 6.0 1.0 6.0 #> 229 flat 3 17.0 12.0 5.0 9.0 1.0 40.0 50.0 0.0 18.0 0.0 0.0 4.0 #> 230 flat 2 13.0 8.0 3.0 8.0 2.0 33.0 34.0 5.0 11.0 4.0 1.0 10.0 #> 231 flat 2 17.0 4.0 7.0 8.0 3.0 25.0 36.0 8.0 12.5 3.0 3.0 11.0 #> 232 flat 2 13.0 11.0 3.0 9.0 0.0 49.0 54.0 3.0 18.0 6.0 7.0 6.0 #> 233 flat 4 21.0 6.0 8.0 11.0 2.0 28.0 32.0 2.0 6.0 3.0 0.0 9.0 #> 234 flat 1 6.0 8.0 2.0 4.0 4.0 46.0 49.0 7.6 12.9 7.5 3.8 1.0 #> 235 flat 1 10.0 18.0 4.0 5.0 1.0 54.0 39.0 1.0 12.0 0.0 1.0 0.0 #> 236 flat 1 19.0 11.0 6.0 10.0 2.0 36.0 33.0 13.0 15.0 7.0 4.0 13.0 #> 237 flat 1 7.0 18.0 3.0 3.0 2.0 57.0 51.0 2.0 13.5 0.0 2.0 9.0 #> 238 flat 1 8.0 9.0 3.0 3.0 1.0 41.0 36.0 0.0 8.0 1.0 2.0 3.0 #> 239 flat 2 9.0 6.0 1.0 6.0 2.0 36.0 33.0 13.0 8.0 8.0 2.0 23.0 #> 240 flat 2 10.0 6.0 2.0 6.0 3.0 38.0 59.0 17.0 16.0 12.0 16.0 14.0 #> 241 flat 2 14.0 14.0 5.0 8.0 0.0 39.0 38.0 0.0 11.0 0.0 1.0 2.0 #> 242 flat 2 11.0 4.0 3.0 6.0 2.0 29.0 47.0 17.0 12.0 13.0 5.0 24.0 #> 243 flat 2 6.0 3.0 1.0 4.0 5.0 29.0 34.0 13.0 15.0 10.0 2.0 11.0 #> 244 flat 2 10.0 11.0 3.0 5.0 2.0 45.0 51.0 1.0 14.0 0.0 3.0 4.0 #> 245 flat 3 6.0 10.0 2.0 2.0 1.0 49.0 46.0 2.0 17.0 4.0 5.0 4.0 #> 246 flat 3 10.0 11.0 2.0 7.0 3.0 36.0 44.0 0.0 17.0 5.0 8.0 6.0 #> 247 flat 2 13.0 10.0 4.0 8.0 2.0 44.0 39.0 9.0 13.0 7.0 1.0 11.0 #> 248 flat 2 13.0 9.0 3.0 10.0 1.0 29.0 45.0 10.0 15.0 11.0 3.0 13.0 #> 249 flat 3 2.0 20.0 1.0 0.0 1.0 41.0 45.0 7.0 10.0 4.0 2.0 5.0 #> 250 flat 3 11.7 3.0 4.0 6.7 6.0 27.0 38.0 12.0 12.0 3.0 3.0 15.0 #> 251 flat 2 11.0 5.0 3.0 8.0 3.0 32.0 34.0 9.0 6.0 5.0 0.0 22.0 #> 252 flat 2 11.0 12.0 4.0 6.0 2.0 51.0 48.0 1.0 15.0 1.0 4.0 7.0 #> 253 flat 2 21.0 5.0 7.0 11.0 1.0 40.0 49.0 0.0 18.0 3.0 11.0 19.0 #> 254 flat 3 2.0 22.0 2.0 0.0 4.0 71.0 65.0 3.0 24.5 0.0 11.0 0.0 #> 255 flat 3 18.0 8.0 5.0 11.0 1.0 26.0 50.0 0.0 18.0 0.0 0.0 3.0 #> 256 flat 3 18.0 17.0 7.0 8.0 2.0 43.0 28.0 17.0 14.0 18.0 4.0 19.0 #> 257 flat 3 20.0 11.0 6.0 12.0 1.0 27.0 22.0 22.0 16.5 25.0 0.0 21.0 #> 258 flat 4 19.0 5.0 6.0 11.0 2.0 35.0 36.0 6.0 12.0 6.0 1.0 15.0 #> 259 flat 4 16.0 17.0 6.0 10.0 2.0 44.0 38.0 16.0 12.0 14.0 4.0 21.0 #> 260 flat 4 14.0 13.0 5.0 7.0 2.0 37.0 50.0 0.0 15.0 3.0 6.0 6.0 #> 261 flat 4 16.0 16.0 4.0 9.0 1.0 55.0 56.0 17.0 14.0 15.0 15.0 19.0 #> 262 flat 4 9.0 6.0 1.0 7.0 2.0 38.0 39.0 2.0 5.0 4.0 1.0 5.0 #> 263 flat 4 18.0 9.0 5.0 13.0 5.0 36.0 26.0 21.0 12.0 19.0 0.0 25.0 #> 264 flat 3 10.0 10.0 3.0 5.0 1.0 35.0 37.0 13.0 11.0 11.0 0.0 0.0 #> 265 flat 4 17.0 6.0 6.0 11.0 4.0 29.0 46.0 7.0 9.0 13.0 7.0 7.0 #> 266 flat 4 15.0 16.0 7.0 7.0 3.0 50.0 36.0 19.0 10.0 25.0 8.0 16.0 #> 267 flat 3 15.0 12.0 6.0 7.0 1.0 49.0 42.0 5.0 12.0 9.0 2.0 16.0 #> 268 flat 3 7.7 13.0 3.0 3.7 4.0 37.0 35.0 17.0 12.5 11.0 2.0 18.0 #> 269 flat 3 16.0 23.0 4.0 10.0 3.0 58.0 53.0 6.0 18.0 5.0 8.0 10.0 #> 270 flat 1 10.0 4.0 2.0 7.0 3.0 30.0 31.0 8.0 14.0 15.0 4.0 8.0 #> 271 flat 1 2.0 5.0 0.0 0.0 5.0 54.0 50.0 12.0 19.0 8.0 5.0 13.0 #> 272 flat 1 13.0 1.0 2.0 11.0 5.0 24.0 20.0 29.0 10.0 29.0 0.0 23.0 #> 273 flat 4 9.0 16.0 5.0 2.0 2.0 62.0 32.0 12.0 8.0 10.0 2.0 15.0 #> 274 flat 4 16.0 11.0 5.0 9.0 2.0 32.0 38.0 3.0 10.0 2.0 1.0 11.0 #> 275 flat 4 17.0 4.0 8.0 7.0 1.0 27.0 21.0 19.0 4.0 15.0 0.0 22.0 #> 276 flat 4 15.0 16.0 5.0 8.0 4.0 36.0 54.0 0.0 13.0 13.0 5.0 4.0 #> 277 flat 3 13.0 20.0 2.0 9.0 3.0 46.0 45.0 0.0 14.0 1.0 3.0 9.0 #> 278 flat 4 17.0 23.0 6.0 10.0 2.0 38.0 41.0 19.0 18.0 13.0 6.0 10.0 #> 279 flat 4 0.0 17.0 0.0 0.0 0.0 51.0 54.0 1.0 15.0 3.0 5.0 13.0 #> 280 flat 4 10.0 21.0 5.0 5.0 2.0 58.0 42.0 5.0 15.0 6.0 3.0 6.0 #> 281 flat 4 13.0 1.0 4.0 9.0 2.0 26.0 32.0 1.0 8.0 8.0 0.0 10.0 #> 282 flat 3 16.0 8.0 3.0 12.0 2.0 35.0 41.0 11.0 14.0 11.0 9.0 3.0 #> 283 flat 3 19.0 12.0 7.0 9.0 1.0 32.0 48.0 16.0 20.0 13.0 9.0 13.0 #> 284 flat 3 16.0 14.0 5.0 11.0 4.0 39.0 60.0 5.0 15.0 4.0 4.0 6.0 #> 285 flat 1 8.0 13.0 2.0 4.0 1.0 44.0 41.0 6.0 11.0 10.0 7.0 8.0 #> 286 flat 1 10.0 11.0 2.0 6.0 2.0 40.6 46.0 3.0 14.0 0.0 4.0 0.0 #> 287 flat 1 21.0 12.0 7.0 11.0 1.0 43.0 38.0 1.0 10.0 4.0 1.0 2.0 #> 288 flat 1 12.0 12.0 2.0 9.0 2.0 35.0 43.0 3.0 15.0 11.0 6.0 2.0 #> 289 flat 1 16.0 13.0 5.0 11.0 3.0 52.0 55.0 0.0 18.0 0.0 7.0 2.0 #> 290 flat 2 14.0 10.0 4.0 9.0 2.0 53.0 44.0 5.0 10.0 7.0 6.0 10.0 #> 291 flat 2 4.0 12.0 3.0 2.0 4.0 51.0 54.0 10.0 19.0 9.0 12.0 11.0 #> 292 flat 1 18.0 12.0 7.0 9.0 2.0 37.0 31.0 23.0 13.0 25.0 1.0 15.0 #> 293 flat 1 13.0 8.0 3.0 7.0 1.0 32.0 28.0 7.0 2.5 4.0 2.0 3.0 #> 294 flat 1 5.0 4.0 0.0 3.0 0.0 34.0 43.0 0.0 11.0 1.0 7.0 11.0 #> 295 flat 2 10.0 7.0 4.0 6.0 5.0 38.0 31.0 10.0 13.0 1.0 0.0 11.0 #> 296 flat 2 17.0 11.0 7.0 8.0 5.0 32.0 33.0 15.0 12.0 21.0 6.0 22.0 #> 297 flat 4 21.0 12.0 7.0 12.0 0.0 46.0 38.0 10.0 10.0 8.0 0.0 11.0 #> 298 flat 2 3.0 7.0 1.0 1.0 7.0 36.0 46.0 11.0 14.0 15.0 2.0 8.0 #> 299 flat 2 15.0 7.0 3.0 10.0 2.0 43.0 41.0 0.0 12.0 0.0 0.0 7.0 #> 300 flat 2 19.0 9.0 7.0 11.0 2.0 47.0 48.0 0.0 15.0 0.0 1.0 3.0 #> 301 flat 2 16.0 4.0 5.0 9.0 2.0 28.0 40.0 7.0 13.0 7.0 2.0 14.0 #> 302 flat 4 19.0 5.0 6.0 11.0 3.0 23.0 29.0 22.0 16.0 17.0 0.0 24.0 #> 303 flat 4 15.0 15.0 7.0 7.0 1.0 55.0 62.0 0.0 16.0 0.0 16.0 0.0 #> 304 flat 4 4.0 1.0 2.0 1.0 7.0 28.0 35.0 9.0 11.0 10.0 0.0 17.0 #> 305 flat 2 6.0 8.0 2.0 2.0 2.0 30.0 60.0 0.0 20.0 8.0 10.0 0.0 #> 306 flat 2 17.0 6.0 5.0 11.0 2.0 25.0 24.0 11.0 10.0 7.0 0.0 11.0 #> 307 flat 2 12.0 11.0 4.0 6.0 3.0 34.0 33.0 5.0 10.0 8.0 1.4 17.0 #> 308 flat 3 5.0 21.0 3.0 2.0 3.0 61.0 54.0 2.0 18.0 0.0 7.0 3.0 #> 309 flat 3 15.0 3.0 4.0 11.0 3.0 26.0 25.0 13.0 10.0 7.0 2.0 17.0 #> 310 flat 3 11.0 4.0 2.0 7.0 1.0 33.0 54.0 15.0 10.0 18.0 7.3 6.7 #> 311 flat 3 17.0 7.0 5.0 10.0 0.0 26.0 39.0 7.0 14.0 9.0 2.0 0.0 #> 312 flat 1 14.0 7.0 3.0 10.0 5.0 35.0 34.0 11.0 11.0 9.0 1.0 6.0 #> 313 flat 1 20.0 14.0 8.0 9.0 0.0 41.0 39.0 2.0 10.0 7.0 2.0 1.0 #> 314 flat 1 18.0 17.0 6.0 9.0 3.0 53.0 52.0 3.0 13.0 7.0 7.0 7.0 #> 315 flat 1 10.0 5.0 4.0 5.0 1.0 35.0 38.0 13.0 15.0 11.0 2.0 7.0 #> 316 flat 2 13.0 11.0 5.0 5.0 0.0 35.0 33.0 16.0 6.0 12.0 0.0 13.0 #> 317 flat 4 17.0 0.0 6.0 10.0 2.0 31.0 34.0 18.0 11.0 18.0 0.0 16.0 #> 318 flat 4 11.0 8.0 1.0 10.0 2.0 38.0 34.0 8.0 8.0 5.0 2.0 6.0 #> 319 flat 4 15.0 16.0 5.0 9.0 2.0 40.0 59.0 8.0 24.0 1.0 8.0 4.0 #> 320 flat 4 14.0 8.0 6.0 7.0 1.0 37.0 45.0 0.0 12.0 1.0 4.0 1.0 #> 321 flat 4 16.0 5.0 5.0 9.0 2.0 30.0 25.0 4.0 12.0 1.0 0.4 10.7 #> 322 flat 4 11.0 9.0 3.0 7.0 2.0 31.0 53.0 8.0 15.5 1.0 6.0 21.0 #> 323 flat 4 16.0 6.0 5.0 8.0 2.0 33.0 41.0 1.0 9.0 5.0 2.0 2.0 #> 324 flat 4 11.0 5.0 4.0 5.0 2.0 30.0 40.0 0.0 10.0 0.0 1.0 5.0 #> 325 flat 1 14.0 1.0 3.0 9.0 5.0 27.0 21.0 19.0 9.0 20.0 0.0 21.0 #> 326 flat 1 17.0 16.0 5.0 10.0 1.0 59.0 38.0 0.0 7.0 1.0 7.0 0.0 #> 327 flat 3 18.0 6.0 8.0 9.0 1.0 37.0 43.0 8.0 8.0 1.0 2.0 20.0 #> 328 flat 3 16.0 22.0 8.0 6.0 1.0 58.0 41.0 8.0 15.0 0.0 0.0 9.0 #> 329 flat 3 8.0 10.0 1.0 6.0 2.0 39.0 46.0 5.0 16.0 0.0 6.0 3.0 #> 330 flat 3 14.0 7.0 2.0 10.0 2.0 35.0 38.0 1.0 10.0 3.0 0.0 4.0 #> TA2 PA2 NA2 state2 MEQ BDI #> 1 5.0 7.0 4.0 NA NA 0.04761905 #> 2 14.0 5.0 5.0 NA NA 0.33333333 #> 3 15.0 3.0 1.0 NA NA 0.19047619 #> 4 15.0 0.0 2.0 NA NA 0.38461538 #> 5 15.0 16.0 13.0 NA NA 0.38095238 #> 6 12.0 2.0 2.0 NA NA 0.23809524 #> 7 14.0 8.0 8.0 NA NA 0.30769231 #> 8 15.0 16.0 0.0 NA NA 0.00000000 #> 9 6.0 14.0 0.0 NA NA 0.00000000 #> 10 19.0 11.0 2.0 NA NA 0.33333333 #> 11 21.0 3.0 7.0 NA NA 0.38095238 #> 12 11.0 29.0 0.0 NA NA 0.14285714 #> 13 17.0 4.0 0.0 NA NA 0.00000000 #> 14 18.0 21.0 2.0 NA NA 0.00000000 #> 15 18.0 12.0 4.0 NA NA 0.04761905 #> 16 26.0 21.0 9.0 NA NA 0.19047619 #> 17 17.0 4.0 18.0 NA NA 0.66666667 #> 18 20.0 21.0 6.0 NA NA 0.33333333 #> 19 22.0 12.0 12.0 NA NA 0.42857143 #> 20 10.0 8.0 1.0 NA NA 0.00000000 #> 21 17.0 7.0 2.0 NA NA 0.04761905 #> 22 22.0 12.0 4.0 NA NA 0.52380952 #> 23 25.0 2.0 6.0 NA NA 0.04761905 #> 24 19.0 7.0 1.0 NA NA 0.42857143 #> 25 15.0 15.0 3.0 NA NA 0.28571429 #> 26 17.0 10.0 6.0 NA NA 0.47619048 #> 27 9.0 10.0 0.0 NA NA 0.09523810 #> 28 13.0 16.0 0.0 NA NA 0.09523810 #> 29 14.0 5.0 0.0 NA NA 0.42857143 #> 30 11.0 24.0 0.0 NA NA 0.38095238 #> 31 19.0 15.0 10.0 NA NA 0.04761905 #> 32 13.0 14.0 5.0 NA NA 0.80952381 #> 33 17.0 15.0 1.0 NA NA 0.23809524 #> 34 13.0 22.0 1.0 NA NA 0.28571429 #> 35 23.0 13.0 11.0 NA NA 0.38095238 #> 36 20.0 8.0 4.0 NA NA 0.33333333 #> 37 8.0 0.0 1.0 NA NA 0.23809524 #> 38 10.0 2.0 0.0 NA NA 0.38095238 #> 39 13.0 21.0 0.0 NA NA 0.04761905 #> 40 11.0 2.0 1.0 NA NA 0.80952381 #> 41 22.0 10.0 14.0 NA NA 0.19047619 #> 42 16.0 11.0 1.0 NA NA 0.00000000 #> 43 19.0 15.0 20.0 NA NA 0.14285714 #> 44 9.0 12.0 10.0 NA NA 0.38095238 #> 45 17.0 24.0 0.0 NA NA 0.09523810 #> 46 12.0 1.0 6.0 NA NA 0.42857143 #> 47 24.0 5.0 15.0 NA NA 0.85714286 #> 48 16.0 7.0 11.0 NA NA 0.66666667 #> 49 25.0 14.0 16.0 NA NA 0.61904762 #> 50 16.0 8.0 6.0 NA NA 0.09523810 #> 51 18.0 15.0 1.0 NA NA 0.42857143 #> 52 21.0 4.0 7.0 NA NA 0.52380952 #> 53 19.0 6.0 6.0 NA NA 0.23809524 #> 54 9.0 0.0 0.0 NA NA 0.33333333 #> 55 9.0 9.0 2.0 NA NA 0.09523810 #> 56 9.0 15.0 0.0 NA NA 0.04761905 #> 57 16.0 13.0 4.0 NA NA 0.33333333 #> 58 8.0 8.0 0.0 NA NA 0.09523810 #> 59 13.0 18.0 0.0 NA NA 0.28571429 #> 60 16.0 10.0 5.0 NA NA 0.28571429 #> 61 10.0 11.0 0.0 NA NA 0.04761905 #> 62 14.0 23.0 0.0 NA NA 0.19047619 #> 63 22.0 6.0 5.0 NA NA 0.85714286 #> 64 18.0 11.0 3.0 NA NA 0.19047619 #> 65 10.0 16.0 0.0 NA NA 0.14285714 #> 66 13.0 14.0 0.0 NA NA 1.14285714 #> 67 11.0 16.0 1.0 NA NA 0.28571429 #> 68 12.0 20.0 0.0 NA NA 0.23809524 #> 69 8.0 12.0 4.0 NA NA 0.00000000 #> 70 15.0 2.0 3.0 NA NA 0.42857143 #> 71 16.0 17.0 4.0 NA NA 0.57142857 #> 72 19.0 16.0 1.0 NA NA 0.47619048 #> 73 21.0 17.0 8.0 NA NA 0.04761905 #> 74 6.0 10.0 2.0 NA NA 0.00000000 #> 75 11.0 20.0 0.0 NA NA 0.14285714 #> 76 20.0 9.0 18.0 NA NA 1.28571429 #> 77 15.0 8.0 1.0 NA NA 0.38095238 #> 78 18.0 3.0 0.0 NA NA 0.33333333 #> 79 16.0 15.0 3.0 NA NA 0.76190476 #> 80 12.0 13.0 0.0 NA NA 0.00000000 #> 81 15.0 8.0 1.0 NA NA 0.14285714 #> 82 15.0 16.0 0.0 NA NA 0.76190476 #> 83 14.0 15.0 0.0 NA NA 0.09523810 #> 84 22.0 6.0 10.0 NA NA 0.61904762 #> 85 7.0 13.0 0.0 NA NA 0.00000000 #> 86 21.0 3.0 6.0 NA NA 0.04761905 #> 87 16.0 11.0 7.0 NA NA 0.33333333 #> 88 19.0 6.0 7.0 NA NA 0.14285714 #> 89 22.0 6.0 15.0 NA NA 1.00000000 #> 90 9.0 8.0 1.0 NA NA 0.14285714 #> 91 3.0 5.0 0.0 NA NA 0.52380952 #> 92 6.0 9.0 3.0 NA NA 0.23809524 #> 93 30.0 14.0 14.0 NA NA 1.04761905 #> 94 21.0 0.0 3.0 NA NA 0.42857143 #> 95 13.0 17.0 7.0 NA NA 0.20000000 #> 96 25.0 12.0 30.0 NA NA 1.23809524 #> 97 14.0 9.0 1.0 NA NA 0.66666667 #> 98 29.0 4.0 13.0 NA NA 0.47619048 #> 99 26.0 11.0 19.0 NA NA 0.47619048 #> 100 18.0 10.0 9.0 NA NA 0.14285714 #> 101 14.0 16.0 2.0 NA NA 0.38095238 #> 102 20.0 4.0 5.0 NA NA 0.14285714 #> 103 23.0 1.0 7.0 NA NA 0.09523810 #> 104 16.0 15.0 7.0 NA NA 0.23809524 #> 105 15.0 17.0 0.0 NA NA 0.28571429 #> 106 14.0 12.0 0.0 NA NA 0.23809524 #> 107 15.0 14.0 6.0 NA NA 0.61904762 #> 108 9.0 7.0 0.0 NA NA 0.14285714 #> 109 14.0 1.0 0.0 NA NA 0.00000000 #> 110 15.0 0.0 1.0 NA NA 0.05000000 #> 111 12.0 3.0 0.0 NA NA 0.09523810 #> 112 11.0 5.0 0.0 NA NA 0.14285714 #> 113 20.0 17.0 6.0 NA NA 0.09523810 #> 114 21.0 3.0 15.0 NA NA 0.14285714 #> 115 19.0 24.0 10.0 NA NA 0.00000000 #> 116 20.0 1.0 10.0 NA NA 0.09523810 #> 117 25.0 5.0 9.0 NA NA 0.14285714 #> 118 9.0 8.0 0.0 NA NA 0.00000000 #> 119 21.0 21.0 19.0 NA NA 0.14285714 #> 120 21.0 12.0 10.0 NA NA 0.33333333 #> 121 16.0 3.0 3.0 NA NA 0.15384615 #> 122 22.0 2.0 5.0 NA NA 0.61904762 #> 123 13.0 11.0 10.0 NA NA 0.14285714 #> 124 20.0 15.0 4.0 NA NA 0.33333333 #> 125 27.0 23.0 16.0 NA NA 0.14285714 #> 126 26.0 5.0 9.0 NA NA 0.09523810 #> 127 24.0 9.0 6.0 NA NA 0.28571429 #> 128 16.0 12.0 4.0 NA NA 0.19047619 #> 129 18.0 2.0 4.0 NA NA 0.09523810 #> 130 11.0 10.0 1.0 NA NA 0.19047619 #> 131 18.0 3.0 10.0 NA NA 1.46153846 #> 132 16.0 24.0 5.0 NA NA 0.19047619 #> 133 13.0 6.0 0.0 NA NA 0.04761905 #> 134 12.0 11.0 0.0 NA NA 0.09523810 #> 135 14.0 15.0 2.0 NA NA 0.14285714 #> 136 11.0 12.0 5.0 NA NA 0.14285714 #> 137 22.0 12.0 11.0 NA NA 0.38095238 #> 138 15.0 3.0 5.0 NA NA 1.04761905 #> 139 15.0 9.0 4.0 NA NA 0.00000000 #> 140 8.0 8.0 2.0 NA NA 0.19047619 #> 141 21.0 6.0 10.0 NA NA 0.66666667 #> 142 23.0 4.0 10.0 NA NA 0.38095238 #> 143 21.0 17.0 14.0 NA NA 0.14285714 #> 144 18.0 17.0 15.0 NA NA 0.47619048 #> 145 26.0 6.0 11.0 NA NA 0.33333333 #> 146 13.0 3.0 0.0 NA NA 0.28571429 #> 147 20.0 12.0 8.0 NA NA 0.33333333 #> 148 16.0 16.0 3.0 NA NA 0.38095238 #> 149 12.0 18.0 0.0 NA NA 0.66666667 #> 150 15.0 0.0 0.0 NA NA 0.33333333 #> 151 16.0 0.0 0.0 NA NA 0.00000000 #> 152 12.0 21.0 4.0 NA NA 0.00000000 #> 153 21.0 11.0 2.0 NA NA 0.19047619 #> 154 16.0 2.0 0.0 NA NA 0.47619048 #> 155 15.0 6.0 3.0 NA NA 0.52380952 #> 156 14.0 7.0 1.0 NA NA 0.23809524 #> 157 9.0 20.0 1.0 NA NA 0.47619048 #> 158 17.0 1.0 1.0 NA NA 0.42857143 #> 159 25.0 20.0 11.0 NA NA 0.33333333 #> 160 17.0 5.0 2.0 NA NA 0.52380952 #> 161 15.0 3.0 0.0 48.0 49 NA #> 162 17.0 0.0 2.0 55.0 43 NA #> 163 21.0 14.5 15.0 51.0 50 NA #> 164 21.0 12.0 20.0 71.0 49 NA #> 165 8.0 0.0 3.0 35.0 39 NA #> 166 17.0 0.0 0.0 37.0 21 NA #> 167 15.0 10.0 0.0 37.0 39 NA #> 168 16.0 15.0 0.0 28.0 49 NA #> 169 11.0 3.0 0.0 36.0 33 NA #> 170 14.6 7.5 0.0 36.0 26 NA #> 171 12.0 7.0 1.0 39.0 38 NA #> 172 8.0 16.0 0.0 26.0 44 NA #> 173 13.0 6.0 1.0 37.0 55 NA #> 174 18.0 12.0 2.0 49.0 26 NA #> 175 12.0 6.0 4.0 38.0 21 NA #> 176 22.0 5.0 6.0 52.0 47 NA #> 177 21.0 4.0 6.0 46.0 36 NA #> 178 20.0 14.0 6.0 45.0 28 NA #> 179 20.0 5.0 7.0 51.0 41 NA #> 180 19.0 0.0 12.0 60.0 41 NA #> 181 21.0 11.0 13.0 58.0 49 NA #> 182 21.0 5.0 6.0 39.0 50 NA #> 183 7.0 6.0 0.0 28.0 39 NA #> 184 14.0 6.0 5.0 46.0 33 NA #> 185 8.0 5.0 1.0 40.0 48 NA #> 186 12.0 5.0 4.0 40.0 43 NA #> 187 9.0 10.0 5.5 38.0 35 NA #> 188 22.0 8.0 4.0 44.0 41 NA #> 189 21.0 3.0 3.0 49.0 52 NA #> 190 15.0 2.0 4.0 50.0 45 NA #> 191 21.5 1.0 6.0 54.0 40 NA #> 192 12.0 3.0 8.0 47.0 49 NA #> 193 16.0 11.0 1.0 38.0 57 NA #> 194 16.0 9.0 5.0 48.0 52 NA #> 195 14.0 16.0 2.0 30.0 56 NA #> 196 10.0 14.0 2.0 35.0 28 NA #> 197 10.0 4.0 7.0 44.0 30 NA #> 198 17.0 10.0 7.0 52.0 47 NA #> 199 15.0 3.0 8.0 56.0 36 NA #> 200 16.0 2.0 7.0 60.0 37 NA #> 201 18.0 1.0 19.0 60.0 48 NA #> 202 16.4 7.0 1.9 38.0 38 NA #> 203 12.0 1.0 0.0 33.0 55 NA #> 204 11.0 23.0 7.0 29.0 33 NA #> 205 17.0 10.0 5.0 52.0 38 NA #> 206 14.0 8.0 2.0 43.0 30 NA #> 207 14.0 1.0 0.0 34.0 42 NA #> 208 16.0 16.0 0.0 26.0 20 NA #> 209 13.5 11.0 1.0 31.0 37 NA #> 210 9.0 9.0 0.0 26.0 35 NA #> 211 19.0 2.0 3.0 49.0 30 NA #> 212 16.0 0.0 3.0 44.0 23 NA #> 213 23.0 6.0 12.0 61.0 43 NA #> 214 10.0 2.0 2.0 39.0 32 NA #> 215 19.0 16.0 13.0 37.0 42 NA #> 216 20.0 19.0 12.0 57.0 37 NA #> 217 14.0 5.0 6.0 46.0 41 NA #> 218 19.0 2.0 10.0 56.0 36 NA #> 219 21.0 6.0 9.0 46.5 29 NA #> 220 14.0 5.0 0.0 35.0 43 NA #> 221 15.0 10.0 3.0 46.0 16 NA #> 222 9.0 0.0 0.0 41.0 29 NA #> 223 16.0 20.0 11.0 38.0 38 NA #> 224 14.0 1.0 0.0 43.0 48 NA #> 225 11.0 12.0 1.0 27.0 45 NA #> 226 9.0 4.0 5.0 51.0 44 NA #> 227 16.0 6.0 0.0 49.0 52 NA #> 228 14.0 3.0 0.0 43.0 37 NA #> 229 18.0 2.0 0.0 50.0 15 NA #> 230 11.0 12.0 2.0 30.6 31 NA #> 231 12.5 2.0 0.0 41.0 49 NA #> 232 23.0 7.0 8.0 57.0 36 NA #> 233 9.0 8.0 0.0 24.0 45 NA #> 234 18.0 7.0 12.0 56.0 32 NA #> 235 12.0 0.0 2.0 44.0 26 NA #> 236 16.0 8.0 3.0 34.0 35 NA #> 237 17.5 2.0 15.0 51.0 65 NA #> 238 10.0 1.0 4.0 42.0 22 NA #> 239 15.0 26.0 0.0 28.0 28 NA #> 240 23.0 10.0 18.0 65.0 57 NA #> 241 16.0 0.0 1.0 40.0 32 NA #> 242 25.0 10.0 7.0 49.0 32 NA #> 243 16.0 6.0 4.0 31.0 46 NA #> 244 20.0 2.0 4.0 48.0 48 NA #> 245 18.0 4.0 8.0 53.0 34 NA #> 246 11.0 11.0 3.0 42.0 24 NA #> 247 18.0 9.0 5.0 46.6 46 NA #> 248 20.0 15.0 9.0 53.0 41 NA #> 249 8.0 3.0 0.0 40.0 36 NA #> 250 14.0 11.0 0.0 31.0 17 NA #> 251 23.0 9.0 4.0 43.0 43 NA #> 252 20.0 0.0 2.0 50.0 32 NA #> 253 31.0 12.0 15.0 67.0 35 NA #> 254 14.0 0.0 0.0 50.0 38 NA #> 255 18.0 0.0 0.0 50.0 58 NA #> 256 10.0 14.0 1.0 24.0 30 NA #> 257 16.5 26.0 0.0 22.0 27 NA #> 258 14.0 10.0 0.0 27.0 30 NA #> 259 15.0 18.0 4.0 34.0 43 NA #> 260 15.0 10.0 4.0 50.0 35 NA #> 261 16.0 17.0 6.0 44.0 48 NA #> 262 10.0 8.0 0.0 39.0 47 NA #> 263 16.0 22.0 0.0 24.0 32 NA #> 264 8.0 3.0 0.0 37.0 44 NA #> 265 16.0 14.0 5.0 45.0 41 NA #> 266 15.0 22.0 12.0 44.0 37 NA #> 267 14.0 15.0 1.0 34.0 54 NA #> 268 13.5 14.0 0.0 32.0 29 NA #> 269 18.0 14.0 13.0 46.0 42 NA #> 270 14.0 10.0 7.0 33.0 23 NA #> 271 19.0 6.0 4.0 48.0 16 NA #> 272 12.0 20.0 1.0 23.0 65 NA #> 273 6.0 15.0 1.0 35.0 53 NA #> 274 15.0 10.0 1.0 31.0 46 NA #> 275 10.0 14.0 0.0 20.0 38 NA #> 276 12.7 12.8 3.0 49.0 48 NA #> 277 14.1 1.0 0.5 36.0 35 NA #> 278 18.0 12.0 6.0 46.0 46 NA #> 279 17.0 5.0 0.0 37.8 49 NA #> 280 10.0 7.0 1.0 40.0 41 NA #> 281 13.0 8.0 0.0 30.0 45 NA #> 282 11.0 8.0 8.0 40.0 21 NA #> 283 10.0 8.0 1.0 31.0 48 NA #> 284 14.0 10.0 0.0 37.0 32 NA #> 285 17.0 3.0 9.0 48.0 40 NA #> 286 20.0 1.0 8.0 63.0 27 NA #> 287 11.0 2.0 2.0 39.0 28 NA #> 288 17.0 7.0 11.0 50.0 29 NA #> 289 20.0 1.6 10.3 62.0 27 NA #> 290 13.0 9.0 5.0 42.0 38 NA #> 291 21.0 10.0 12.0 55.8 45 NA #> 292 17.0 12.0 14.0 45.0 46 NA #> 293 12.5 1.0 6.0 48.0 37 NA #> 294 17.0 11.0 20.0 65.0 43 NA #> 295 22.0 0.0 2.0 44.0 33 NA #> 296 18.0 24.0 7.0 25.0 39 NA #> 297 11.0 7.0 0.0 36.0 24 NA #> 298 18.0 3.0 9.0 56.0 58 NA #> 299 18.0 0.0 0.0 46.0 23 NA #> 300 16.0 0.0 5.0 51.0 27 NA #> 301 23.0 9.0 6.0 53.0 54 NA #> 302 17.0 17.7 0.0 29.0 59 NA #> 303 14.0 0.0 7.0 54.0 33 NA #> 304 14.0 9.0 0.0 39.0 45 NA #> 305 19.0 7.0 8.0 57.0 30 NA #> 306 28.0 2.0 15.0 66.0 50 NA #> 307 21.0 11.0 3.0 36.0 55 NA #> 308 14.0 0.0 2.0 52.0 42 NA #> 309 10.0 15.0 0.0 20.0 59 NA #> 310 19.9 14.0 14.1 51.4 50 NA #> 311 15.0 7.0 4.0 38.0 44 NA #> 312 15.0 4.0 6.0 52.0 45 NA #> 313 11.0 5.0 2.0 45.0 42 NA #> 314 24.0 11.0 22.0 61.0 51 NA #> 315 19.0 4.0 10.0 52.0 40 NA #> 316 9.0 7.0 0.0 37.0 34 NA #> 317 10.0 9.0 0.0 33.0 41 NA #> 318 7.0 6.0 0.0 32.0 44 NA #> 319 6.0 0.0 1.0 37.0 42 NA #> 320 14.0 1.0 0.0 39.0 37 NA #> 321 13.0 9.3 0.0 24.0 50 NA #> 322 10.5 17.0 0.0 25.0 22 NA #> 323 11.0 3.0 1.0 40.0 33 NA #> 324 7.0 1.0 0.0 31.0 38 NA #> 325 9.0 17.0 2.0 20.0 57 NA #> 326 14.0 1.0 9.0 46.0 28 NA #> 327 13.0 14.0 1.0 43.0 49 NA #> 328 12.0 0.0 0.0 40.0 42 NA #> 329 17.0 1.0 2.0 45.0 41 NA #> 330 10.0 3.0 0.0 35.0 39 NA ``` ] --- class: inverse, center, middle # Packages --- ##How do we implement machine learning in R? There are  for building and evaluating machine learning models in R. Each implements specific ML models (e.g., {glmnet} for lasso, ridge, and elastic net regularization, {rpart} for decision trees, {randomforest} for random forests). These packages were built by different people over time, so . This can be confusing to remember! -- <img src="data:image/png;base64,#syntax_diff.png" width="70%" height="70%" /> --- ##{caret} Recognizing the need for  the process of building and evaluating machine learning models, Max Kuhn and others developed the `caret` (**C**lassification **A**nd **RE**gression **T**raining) package. This package allows researchers to quickly build and compare many different models. There are 200+ machine learning models available in `caret`. -- `caret` includes functions for: - data visualization - data pre-processing - feature selection - data splitting - model training & testing - variable importance estimation --- ##{caret} .pull-left[ The `train()` function is the primary function for training models and tuning hyperparameters. This  is used to train any and all machine learning models. Users can also specify tuning parameter values and resampling method (e.g., *k*-fold cross-validation) with the `trainControl()` function. ] .pull-right[ <img src="data:image/png;base64,#caret_train.png" width="100%" /> ] -- </br> ```r iris_fit <- train(Species ~., data = iris, method = 'glmnet', trControl = trainControl(method = "cv", number = 10)) ``` --- ##{caret} Because `caret` has historically been the most popular package for machine learning in R, there are many freely available resources, solutions, and answers to questions online. -- .pull-left[ <img src="data:image/png;base64,#caret_tutorial.png" width="100%" /> ] -- .pull-right[ <img src="data:image/png;base64,#appliedpredictivemodeling.png" width="48%" height="48%" /> ] --- ## {tidymodels} The newer {tidymodels} package is the `tidyverse` version of {caret}. Both packages were developed by the same author (Max Kuhn)! {tidymodels} is a  and includes a collection of many packages: - `rsample` for data splitting and resampling - `recipes` for pre-processing - `parsnip` for trying out many models - `workflows` to streamline the pre-processing, modeling, and post-processing - `tune` to optimize model hyperparameters - `yardstick` for model performance metrics - `broom` for converting information to user-friendly formats - `dials` for creating and managing tuning parameters -- We will use the older `caret` but incorporate *aspects* of the newer `tidymodels`<sup>1</sup> - This will give us access to some new features without overwhelming beginners - It will also ease the transition to `tidymodels` if you decide to go that route .footnote[[1] We will use `recipes` and `yardstick` but not `workflows`, `rsample`, `tune`, `parsnip`, or `dials.`] --- class: inverse, center, middle # Exploratory Data Analysis --- ## Typical Workflow <br /> <img src="data:image/png;base64,#workflow.png" width="100%" /> --- ## Typical Workflow <br /> <img src="data:image/png;base64,#workflow_eda.png" width="100%" /> --- ## Exploratory Data Analysis .left-column[ </br> <img src="data:image/png;base64,#explore.jpg" width="100%" /> ] .right-column[ **Goals** - Develop an understanding of your data - Make informed model building decisions (e.g., feature selection) **Questions** - What type of variation occurs in my variables? - Are there any anomalies, errors, or outliers? - How much missing data do I have? - What type of covariation occurs between my variables? - Are there any nonlinearities in my data? - Are my data appropriate for the task? ] -- .right-column[ .bg-light-yellow.b--light-red.ba.bw1.br3.pl4[ **Caution!** Exploratory data analysis should ideally be performed **only** on training data. ] ] --- class: inverse, center, middle # Simple Holdout Set --- ##Why EDA on training data only? The **ultimate goal** of exploratory data analysis is to . We split our data into training and testing subsets to evaluate the accuracy of our model in predicting *unseen* data. This gives us a sense for how our model might perform in the **future** on new datasets. -- </br> If modeling decisions are made based on data patterns we observe in the test set, we  estimates in the test set. **When possible, it is ideal** to perform exploratory data analysis only on your training data only. But note that doesn't mean we can't check the test data for coding errors or data anomalies! --- ##Simple Train/Test Split <img src="data:image/png;base64,#datasplit_simple.png" width="100%" /> --- ##Data Splitting An important note on terminology beyond simple train/test data splits<sup>1</sup>: - **Training**: The data subsample used to explore the data and fit the model. - **Validation**: Used for model evaluation while tuning hyperparameters; often implicitly split via cross-validation. - **Test**: Entirely held-out from model training/tuning; used to provide a unbiased evaluation of the final model. <img src="data:image/png;base64,#datasplit_val.png" width="70%" height="70%" /> .footnote[ [1] More details on model training and cross-validation methods to come tomorrow in section 2-B. ] --- ##Simple Train/Test Split in R Use the `createDataPartition()` function in {caret} to create balanced training and testing splits based on the outcome variable. Random sampling occurs within each factor level to  in the datasets. Specify the proportion of data you want in the training split (e.g., `p = 0.8`) for an 80%/20% data split. Remember to set a seed so your results are ! -- </br> ```r library(caret) set.seed(2021) trainIndex <- createDataPartition(iris$Species, p = .8, list = FALSE, times = 1) ``` --- ##Simple Train/Test Split in R Use the `createDataPartition` row indices to split your data into single train and test sets. ```r irisTrain <- iris[trainIndex, ] irisTest <- iris[-trainIndex, ] ``` -- Now 80% of the data is designated for model training and can be used for exploratory data analysis. 20% of the data is  before testing the model, to avoid overly optimistic results. ```r dim(irisTrain) #> [1] 120 5 dim(irisTest) #> [1] 30 5 ``` --- ##Knowledge check <span style="font-size:30px;">Taylor is interested in building a machine learning model to predict future risk of depression. How should they explore these elements of their dataset?</span> .pull-left[ ### Question 1 **Looking for outliers or data anomalies:** a) Training data only b) Test data only c) Both training and test d) Neither ] .pull-right[ ### Question 2 **Finding features that correlate with the outcome:** a) Training data only b) Test data only c) Both training and test d) Neither ] --- class: inverse, center, middle # Exploratory Data Analysis ## Data Distributions and Error Detection --- ##Data distributions and error detection .left-column[ </br> <img src="data:image/png;base64,#normal_dist.png" width="100%" height="350" /> ] .right-column[ Typically, the first step in exploratory data analysis is to explore data distributions. This provides insight into: - Whether features are normally distributed - Concerning or extreme skewness - Potential data anomalies or errors - Data outliers - Features with low variance - Imbalanced data (categorical variables) Methods: - Summary statistics - Histograms - Bar charts ] --- ##Data distributions and error detection in R The `dfsummary()` function from {summarytools} is useful for quickly identifying trends and anomalies at a glance. -- ```r print(dfSummary(irisTrain), method = 'render') ``` <div class="container st-container"> <div style="max-height:300px;overflow-y:scroll;margin:10px 2px"> <table class="table table-striped table-bordered st-table st-table-striped st-table-bordered st-multiline "> <thead> <tr> <th align="center" class="st-protect-top-border"><strong>No</strong></th> <th align="center" class="st-protect-top-border"><strong>Variable</strong></th> <th align="center" class="st-protect-top-border"><strong>Stats / Values</strong></th> <th align="center" class="st-protect-top-border"><strong>Freqs (% of Valid)</strong></th> <th align="center" class="st-protect-top-border"><strong>Graph</strong></th> <th align="center" class="st-protect-top-border"><strong>Valid</strong></th> <th align="center" class="st-protect-top-border"><strong>Missing</strong></th> </tr> </thead> <tbody> <tr> <td align="center">1</td> <td align="left">Sepal.Length [numeric]</td> <td align="left">Mean (sd) : 5.8 (0.8) min < med < max: 4.4 < 5.8 < 7.7 IQR (CV) : 1.3 (0.1)</td> <td align="left" style="vertical-align:middle">32 distinct values</td> <td align="left" style="vertical-align:middle;padding:0;background-color:transparent"><img style="border:none;background-color:transparent;padding:0" src="data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAAAIMAAABfCAQAAABhaLy5AAAAAmJLR0QA/4ePzL8AAAAHdElNRQflBxMNKwQDFK7TAAABSklEQVR42u3cUW7aQABF0UOVPXYP7XraPWSV7c/7QJECLUkNqPf+ISQYH3mQR2Nz+qX4cu8BPEYxIIb1cv7idJch/Ljy8/T9Hw3r/Gtfbv6UT+zrhfdeDxlBkwKHnQ3XTvx7d9ikeP/EP+a0v1yTAjGsGBDDigExrBgQw/qLy6dHvxI8iOHaEuj+C6Tba1IghhUDYlgxIIYVA2JYMSCGFQNiWDEghhUDYlgxIIYVA2JYMSCGFQNiWDEghhUDYlgPcZfs5S7vpH/OrcRPwHDETnmTAjGsGBDDigExrBgQw4oBMawYEMOKATGsGBDDigExrBgQw4oBMawYEMOKATGsGPBmD/MZHz/+wJjPNoHfbOU+3wPGl/4658+PpkmBGFYMiGHFgBhWDIhhxYAYVgyIYZ3OF2g/n3CFeXvfzlaYp//qyN+tSYEYVgzgNxVtGR5h23/qAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIxLTA3LTE5VDEzOjQzOjA0KzAwOjAwhDMtzgAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMS0wNy0xOVQxMzo0MzowNCswMDowMPVulXIAAAA9dEVYdGljYzpjb3B5cmlnaHQAQ29weXJpZ2h0IDIwMDcgQXBwbGUgSW5jLiwgYWxsIHJpZ2h0cyByZXNlcnZlZC6eZtwpAAAAI3RFWHRpY2M6ZGVzY3JpcHRpb24AR2VuZXJpYyBSR0IgUHJvZmlsZRqnOI4AAAAASUVORK5CYII="></td> <td align="center">120 (100.0%)</td> <td align="center">0 (0.0%)</td> </tr> <tr> <td align="center">2</td> <td align="left">Sepal.Width [numeric]</td> <td align="left">Mean (sd) : 3 (0.4) min < med < max: 2 < 3 < 4.4 IQR (CV) : 0.5 (0.1)</td> <td align="left" style="vertical-align:middle">22 distinct values</td> <td align="left" style="vertical-align:middle;padding:0;background-color:transparent"><img style="border:none;background-color:transparent;padding:0" src="data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAAAIMAAABfCAQAAABhaLy5AAAAAmJLR0QA/4ePzL8AAAAHdElNRQflBxMNKwQDFK7TAAABO0lEQVR42u3cS2rcUBRF0VXBc8wckvEkc/Aok85pCEMCsrEs7L17BfXTgqvGrUc9/ii+ffQXuEcxIIb1dHzwOPniXydvLD/PfsC7dvzyT69+F/D9xHOfP/q6/1NDgRhWDIhhxYAYVgyIYcWAGFYMiGHFgBhWDIhhxYA3b5/Odd+l3aUMZ1Z21y7tGgrEsGJADCsGxLBiQAwrBsSwYkAMKwbEsGJADCsGxLBiQAwrBsSwYkAMKwbEsGJADCsGxLBiQAwrBsSwYsCLs09nj2h9nl4cAbvvIa33raFADCsGXHw89GzXnaq9NcN1N+yGAjGsGBDDigExrBgQw4oBMawYEMOKATGsGBDDigExrBhw813k2U6ucA8L3E/F8Pr/Mm0oEMN6HMfp95f6RfvH4d7w+FJX/s8aCsSwYgB/AaJNFPa2TCg7AAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIxLTA3LTE5VDEzOjQzOjA0KzAwOjAwhDMtzgAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMS0wNy0xOVQxMzo0MzowNCswMDowMPVulXIAAAA9dEVYdGljYzpjb3B5cmlnaHQAQ29weXJpZ2h0IDIwMDcgQXBwbGUgSW5jLiwgYWxsIHJpZ2h0cyByZXNlcnZlZC6eZtwpAAAAI3RFWHRpY2M6ZGVzY3JpcHRpb24AR2VuZXJpYyBSR0IgUHJvZmlsZRqnOI4AAAAASUVORK5CYII="></td> <td align="center">120 (100.0%)</td> <td align="center">0 (0.0%)</td> </tr> <tr> <td align="center">3</td> <td align="left">Petal.Length [numeric]</td> <td align="left">Mean (sd) : 3.7 (1.7) min < med < max: 1 < 4.3 < 6.9 IQR (CV) : 3.5 (0.5)</td> <td align="left" style="vertical-align:middle">39 distinct values</td> <td align="left" style="vertical-align:middle;padding:0;background-color:transparent"><img style="border:none;background-color:transparent;padding:0" src="data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAAAIMAAABfCAQAAABhaLy5AAAAAmJLR0QA/4ePzL8AAAAHdElNRQflBxMNKwQDFK7TAAABSklEQVR42u3cQYoaURhG0WPoPWYPyXqSPfQqk8k3MIRAYSulnXtnIoh18Invr7IuvxRfzn4Dz1EMiGG9XT/4efiL4vvl7Df+8a4P9u3Pp74eeoH3s4/g7rUoEMOKATGsGBDDigExrBgQw4oBf+0pzujHE2zonoDhGTZ0LQrEsGJADCsGxLBiQAwrBsSwYkAMKwbEsGJADCsGPMn06WjHxnW3jOpeiuHIsO62UV2LAjGsGBDDigExrBgQw4oBMawYEMOKATGsGBDDigExrBgQw4oBMayHDuiPXwZ8dg8+T/G4Mwv3rUWBGFYMiGHFgBhWDIhhxYAYVgyIYcWAGFYMeLHrIo91eNhzdRnpJ2S45R/fLQrEsGJADCsGxLBiQAzrxp9Pr3Na7qEM59+b5761KMDl+tN9/M7Cn6FvV1ury3915P+sRYEYVgzgNyD5F47nUYb/AAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIxLTA3LTE5VDEzOjQzOjA0KzAwOjAwhDMtzgAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMS0wNy0xOVQxMzo0MzowNCswMDowMPVulXIAAAA9dEVYdGljYzpjb3B5cmlnaHQAQ29weXJpZ2h0IDIwMDcgQXBwbGUgSW5jLiwgYWxsIHJpZ2h0cyByZXNlcnZlZC6eZtwpAAAAI3RFWHRpY2M6ZGVzY3JpcHRpb24AR2VuZXJpYyBSR0IgUHJvZmlsZRqnOI4AAAAASUVORK5CYII="></td> <td align="center">120 (100.0%)</td> <td align="center">0 (0.0%)</td> </tr> <tr> <td align="center">4</td> <td align="left">Petal.Width [numeric]</td> <td align="left">Mean (sd) : 1.2 (0.8) min < med < max: 0.1 < 1.3 < 2.5 IQR (CV) : 1.5 (0.6)</td> <td align="left" style="vertical-align:middle">22 distinct values</td> <td align="left" style="vertical-align:middle;padding:0;background-color:transparent"><img style="border:none;background-color:transparent;padding:0" src="data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAAAIMAAABfCAQAAABhaLy5AAAAAmJLR0QA/4ePzL8AAAAHdElNRQflBxMNKwQDFK7TAAABPElEQVR42u3cQY7TQBRF0RPUe2QPsB7YQ68SJm/gFmLgdJMYuHdmRYpcR6roy5Xk9kPx6dk3cI1iQAzr5Xjx/fQHxdfbsxdwf8fFvrx96fOpN3p99ko+rDYFYlgxIIYVA2JYMSCGFQN+mSL/ZN8uPKo/kOHKo3qbAjGsGBDDigExrBgQw4oBMawYEMOKATGsGBDDigExrBgQw4oBMawYEMN66HHN2c6ec91/ynVphsedcrUpEMOKATGsGBDDigEXnxvOdnLcOgxb/xTDmXHr7bDVpkAMKwbEsGJADCsGxLBiQAwrBsSwYkAMKwbEsGJADCsGxLBiQAzrnQ/oz//g+Jq9k+H+k4Fr1aZADCsGxLBiQAwrBsSwYkAMKwbEsGJADOt2fGBw/p+F/+a+HL4XefuvVv7b2hSIYcUAfgKm4xT20hudVAAAACV0RVh0ZGF0ZTpjcmVhdGUAMjAyMS0wNy0xOVQxMzo0MzowNCswMDowMIQzLc4AAAAldEVYdGRhdGU6bW9kaWZ5ADIwMjEtMDctMTlUMTM6NDM6MDQrMDA6MDD1bpVyAAAAPXRFWHRpY2M6Y29weXJpZ2h0AENvcHlyaWdodCAyMDA3IEFwcGxlIEluYy4sIGFsbCByaWdodHMgcmVzZXJ2ZWQunmbcKQAAACN0RVh0aWNjOmRlc2NyaXB0aW9uAEdlbmVyaWMgUkdCIFByb2ZpbGUapziOAAAAAElFTkSuQmCC"></td> <td align="center">120 (100.0%)</td> <td align="center">0 (0.0%)</td> </tr> <tr> <td align="center">5</td> <td align="left">Species [factor]</td> <td align="left">1. setosa 2. versicolor 3. virginica</td> <td align="left" style="padding:0;vertical-align:middle"><table style="border-collapse:collapse;border:none;margin:0"><tr style="background-color:transparent"><td style="padding:0 5px 0 7px;margin:0;border:0" align="right">40</td><td style="padding:0 2px 0 0;border:0;" align="left">(</td><td style="padding:0;border:0" align="right">33.3%</td><td style="padding:0 4px 0 2px;border:0" align="left">)</td></tr><tr style="background-color:transparent"><td style="padding:0 5px 0 7px;margin:0;border:0" align="right">40</td><td style="padding:0 2px 0 0;border:0;" align="left">(</td><td style="padding:0;border:0" align="right">33.3%</td><td style="padding:0 4px 0 2px;border:0" align="left">)</td></tr><tr style="background-color:transparent"><td style="padding:0 5px 0 7px;margin:0;border:0" align="right">40</td><td style="padding:0 2px 0 0;border:0;" align="left">(</td><td style="padding:0;border:0" align="right">33.3%</td><td style="padding:0 4px 0 2px;border:0" align="left">)</td></tr></table></td> <td align="left" style="vertical-align:middle;padding:0;background-color:transparent"><img style="border:none;background-color:transparent;padding:0" src="data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAAADcAAABDCAQAAACSYVUaAAAAAmJLR0QA/4ePzL8AAAAHdElNRQflBxMNKwV0E55FAAAAfUlEQVRYw+3WsQ2AMAxEURuxIzuwD+yQKaENSEQoxE5k/nVu/AoXPj3EM5OrFpyb82E3OeSqD5zI0hxLlyn27eDg4ODghuVu/y7VbXkdzR+4Vq8pJRd+dLvNpuNSjeDg4ODgenN0lYahq3wKXQUODg4ObgCuZzWyT+zbOXMnOYsQtmjbE/cAAAAldEVYdGRhdGU6Y3JlYXRlADIwMjEtMDctMTlUMTM6NDM6MDUrMDA6MDAiRCZ6AAAAJXRFWHRkYXRlOm1vZGlmeQAyMDIxLTA3LTE5VDEzOjQzOjA1KzAwOjAwUxmexgAAAD10RVh0aWNjOmNvcHlyaWdodABDb3B5cmlnaHQgMjAwNyBBcHBsZSBJbmMuLCBhbGwgcmlnaHRzIHJlc2VydmVkLp5m3CkAAAAjdEVYdGljYzpkZXNjcmlwdGlvbgBHZW5lcmljIFJHQiBQcm9maWxlGqc4jgAAAABJRU5ErkJggg=="></td> <td align="center">120 (100.0%)</td> <td align="center">0 (0.0%)</td> </tr> </tbody> </table> </div> <p>Generated by <a href='https://github.com/dcomtois/summarytools'>summarytools</a> 0.9.9 (<a href='https://www.r-project.org/'>R</a> version 4.1.0)<br/>2021-07-19</p> </div> --- ##Data distributions and error detection in R Overlaying distributions on the same plot can also be helpful. We can use the `featurePlot()` function in {caret}. -- ```r # basic density plot featurePlot(x = irisTrain[, 1:4], y = irisTrain$Species, plot = 'density') ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-41-1.png" width="60%" style="display: block; margin: auto;" /> .footnote[ credit to https://topepo.github.io/caret/visualizations.html ] --- ##Data distributions and error detection in R Overlaying distributions on the same plot can also be helpful. We can use the `featurePlot()` function in {caret}. ```r # make it prettier: free x & y axis, add legend, change plot character featurePlot(x = irisTrain[, 1:4], y = irisTrain$Species, plot = 'density', scales = list(x = list(relation = "free"), y = list(relation = "free")), adjust = 1.5, pch = "|", auto.key = list(columns = 3)) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-43-1.png" width="60%" style="display: block; margin: auto;" /> .footnote[ credit to https://topepo.github.io/caret/visualizations.html ] --- ##Are there data anomalies, errors, or outliers? Check data distributions and summary statistics for: - Extreme values - Nonsensical values - Inconsistencies - Low variance - You may need to adjust plot margins or axes! .pull-left[ <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-44-1.png" width="100%" /> ] .pull-right[ <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-45-1.png" width="100%" /> ] --- class: inverse, center, middle # Exploratory Data Analysis ## Missing Data --- ##A technical explanation of missing data **Missing completely at random (MCAR)** - No systematic pattern of missing data; the probability of an observation being missing does not depend on any observed or missing data values. - E.g., If a weighing scale sometimes runs out of batter, missing data on weight is only due to bad luck and not any measured or missing data. **Missing at random (MAR)** - Systematic relationship between missing values and the *observed* data, but *not* the missing data. - E.g., If people with eating disorders are more likely to decline being weighed, missing data on weight is systematically related to eating disorder diagnosis. **Missing not at random (MNAR)** - Systematic relationship between missing values and those values themselves. - E.g., If people with higher weights are more likely to decline being weighed, missing data on weight is systematically related to *weight itself*. --- ##An intuitive explanation of missing data <img src="data:image/png;base64,#mcar_dog.png" width="60%" /> --- ##Missing Data .footnote[ credit to https://www.datacamp.com/community/tutorials/visualize-data-vim-package ] .pull-left[ <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-48-1.png" width="100%" /> ] -- .pull-right[ <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-49-1.png" width="100%" /> ] -- The {VIM} package is particularly helpful for visualizing patterns of missing data. Two helpful questions to guide missing data visualization: - Which variables have missing observations (and how many)? - Does missing data in one variable depend on other variables? --- ##Missing data visualization in R **Aggregation plots** are useful for inspecting the prevalence of missing data. ```r aggr(biopics, numbers = TRUE, prop = c(TRUE, FALSE), col = c("bisque2","darkcyan")) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-51-1.png" width="75%" height="75%" style="display: block; margin: auto;" /> .footnote[ credit to https://www.datacamp.com/community/tutorials/visualize-data-vim-package ] --- ##Missing data visualization in R We also want to know if missing data systematically vary by other observed data. If the other data are numeric we use a **spinogram**; if categorical we can use a **spineplot**. ```r spineMiss(biopics[, c("sub_race", "earnings")], col = c("bisque2","darkcyan")) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-52-1.png" width="70%" height="70%" style="display: block; margin: auto;" /> .footnote[ credit to https://www.datacamp.com/community/tutorials/visualize-data-vim-package ] --- ##Missing data visualization in R Let's flip the two variables to ask: does the percentage of missing data in `sub_race` differ by `earnings`? ```r spineMiss(biopics[, c("earnings", "sub_race")], col = c("bisque2","darkcyan")) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-53-1.png" width="75%" height="75%" style="display: block; margin: auto;" /> .footnote[ credit to https://www.datacamp.com/community/tutorials/visualize-data-vim-package ] --- ##Missing data visualization in R **Mosiac plots** generalize spineplots and spinograms (which only plot two variables at a time) to multiple variables. ```r mosaicMiss(biopics[, c("sub_sex", "US_movie", "earnings")], highlight = 3, plotvars = 1:2, miss.labels = FALSE, col = c("bisque2","darkcyan")) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-55-1.png" width="70%" height="70%" style="display: block; margin: auto;" /> .footnote[ credit to https://www.datacamp.com/community/tutorials/visualize-data-vim-package ] --- ##Missing data visualization in R **Parallel coordinate plots** allow us to look at patterns of missingness across the entire dataset. ```r parcoordMiss(biopics, highlight = 'earnings', alpha = 0.6, col = c("bisque2","darkcyan")) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-56-1.png" width="75%" height="75%" style="display: block; margin: auto;" /> .footnote[ credit to https://www.datacamp.com/community/tutorials/visualize-data-vim-package ] --- class: inverse, center, middle # Exploratory Data Analysis ## Feature Covariation and Correlations --- ##Feature covariation and correlations .left-column[ </br> <img src="data:image/png;base64,#correlation_meme.png" width="100%" /> ] .right-column[ In addition to exploring variation *within* features, we should also explore the covariation that occurs (1) *between* features and (2) between features and labels. This provides insight into: - Highly correlated features (multicollinearity) - Potential clusters of features that could be reduced into a single feature - Features with strong relationships to the outcome (feature selection) Methods: - Correlation matrices - Correlation matrices with clustering - Scatterplot matrices ] --- ##Feature covariation and correlations in R ```r data(msq) cormat <- cor(subset(msq, select = c("afraid", "angry", "anxious", "cheerful", "delighted", "depressed", "frustrated", "jittery", "proud", "scared", "unhappy")), use = 'pairwise.complete.obs') corrplot(cormat, tl.col = '#23395b', type = 'lower', tl.cex = 0.8) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-58-1.png" width="100%" /> --- ##Feature covariation and correlations in R ```r # Correlation matrix with numbers corrplot(cormat, type = 'lower', method = 'number', tl.col = '#23395b', tl.cex = 0.8, pch.cex = 0.5, number.cex = 0.55) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-59-1.png" width="100%" /> --- ##Feature covariation and correlations in R ```r # Correlation matrix with hierarchical clustering corrplot(cormat, tl.col = '#23395b', order = 'hclust', addrect = 3, tl.cex = 0.8, pch.cex = 0.5, number.cex = 0.55) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-60-1.png" width="100%" /> --- ##Feature covariation and correlations in R ```r # Scatterplot matrix pairs.panels(irisTrain[, 1:4], method = 'pearson', density = TRUE, ellipses = FALSE, lm = TRUE, cex.cor = 0.8, cex.labels = 0.9) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-61-1.png" width="100%" style="display: block; margin: auto;" /> --- class: inverse, center, middle # Exploratory Data Analysis ## Linearity and Nonlinearity --- ##Linearity and nonlinearity Nonlinearity between features and outcome variables are important to pay attention to, because these data patterns inform algorithm selection. While some algorithms (e.g., decision trees, random forests) can capture and model nonlinearity, other algorithms (e.g., lasso, ridge, elastic net) cannot. The specific **form** of nonlinearity is also important. .pull-left[ <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-62-1.png" width="100%" /> ] .pull-right[ <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-63-1.png" width="100%" /> ] --- ##Linearity and nonlinearity in R We can use the `featurePlot()` function in `caret` to look for nonlinearities between features and outcomes. ```r featurePlot(x = affect[, c(4, 8, 12, 20)], y = affect$NA1, plot = "scatter", type = c("p", "smooth"), labels = c("Feature", "Negative Affect")) ``` <img src="data:image/png;base64,#Day_1B_Slides_files/figure-html/unnamed-chunk-65-1.png" width="80%" style="display: block; margin: auto;" /> --- class: inverse, center, middle # Exploratory Data Analysis ## Takeaways --- ##Using EDA to inform model building <img src="data:image/png;base64,#eda_takeaway.png" width="100%" /> --- class: inverse, center, middle ## Small Group Activity --- ##Small Group Activity We will assign you to a small breakout room. We will jump between rooms to join discussions and answer questions.  Our goal is for everyone to gain experience with all modeling processes, so everyone should work  on their own code, rather than assign one person to share their screen and do the coding. However, please feel free to discuss the topics covered with your group members - we hope you also learn from each other! If you're stuck or have questions, please post them in the chat or workshop Slack channel, and feel free to consult your group members. </br> **BONUS AND HELPFUL FURTHER READING**: - The EDA Chapter in *R for Data Science*: https://r4ds.had.co.nz/exploratory-data-analysis.html - Visualizations in {caret}: https://topepo.github.io/caret/visualizations.html .footnote[ All workshop materials are available on the course website: https://pittmethods.github.io/appliedml/. ]